[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:07:59 dist-test-slave-ck1n krb5kdc[29403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:07:59 dist-test-slave-ck1n krb5kdc[29403](info): set up 2 sockets
Feb 01 03:07:59 dist-test-slave-ck1n krb5kdc[29403](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260201 03:08:00.763975 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.696s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:00 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:00.869674 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:40669
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:40669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:00.934684 29419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:00.934854 29419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:00.934890 29419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:00.936250 29419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:00.936333 29419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:00.936353 29419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:00.936376 29419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:00.936403 29419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:00.937671 29419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:40669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:40669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:00.937896 29419 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:00.938048 29419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:00.940043 29424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:00.940160 29419 server_base.cc:1060] running on GCE node
W20260201 03:08:00.940052 29427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:00.940068 29425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:00.940438 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:00.940652 29419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:00.941816 29419 hybrid_clock.cc:648] HybridClock initialized: now 1769915280941805 us; error 23 us; skew 500 ppm
Feb 01 03:08:00 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915280, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:00.943589 29419 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:00.944058 29419 webserver.cc:492] Webserver started at http://127.28.181.62:42837/ using document root <none> and password file <none>
I20260201 03:08:00.944216 29419 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:00.944262 29419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:00.944376 29419 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:00.945096 29419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "a7841e50f69544b08c06deca9b5c2e47"
format_stamp: "Formatted at 2026-02-01 03:08:00 on dist-test-slave-ck1n"
server_key: "15232d2864c9d177cee08cb4552c53ad"
server_key_iv: "d959a8299400cc30c0276156aa658911"
server_key_version: "encryptionkey@0"
I20260201 03:08:00.945355 29419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "a7841e50f69544b08c06deca9b5c2e47"
format_stamp: "Formatted at 2026-02-01 03:08:00 on dist-test-slave-ck1n"
server_key: "15232d2864c9d177cee08cb4552c53ad"
server_key_iv: "d959a8299400cc30c0276156aa658911"
server_key_version: "encryptionkey@0"
I20260201 03:08:00.946369 29419 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:00.946918 29434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:00.947062 29419 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:00.947152 29419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "a7841e50f69544b08c06deca9b5c2e47"
format_stamp: "Formatted at 2026-02-01 03:08:00 on dist-test-slave-ck1n"
server_key: "15232d2864c9d177cee08cb4552c53ad"
server_key_iv: "d959a8299400cc30c0276156aa658911"
server_key_version: "encryptionkey@0"
I20260201 03:08:00.947211 29419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:00.961031 29419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:00.962323 29419 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:00.962442 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:00.965829 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:40669
I20260201 03:08:00.965889 29486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:40669 every 8 connection(s)
I20260201 03:08:00.966113 29419 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:00.966576 29487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:00.968434 29487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47: Bootstrap starting.
I20260201 03:08:00.968917 29487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:00.969131 29487 log.cc:826] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:00.969672 29487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47: No bootstrap required, opened a new log
I20260201 03:08:00.970621 29487 raft_consensus.cc:359] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } }
I20260201 03:08:00.970713 29487 raft_consensus.cc:385] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:00.970731 29487 raft_consensus.cc:740] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7841e50f69544b08c06deca9b5c2e47, State: Initialized, Role: FOLLOWER
I20260201 03:08:00.970839 29487 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } }
I20260201 03:08:00.970896 29487 raft_consensus.cc:399] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:00.970932 29487 raft_consensus.cc:493] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:00.970973 29487 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:00.971395 29487 raft_consensus.cc:515] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } }
I20260201 03:08:00.971468 29487 leader_election.cc:304] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7841e50f69544b08c06deca9b5c2e47; no voters:
I20260201 03:08:00.971582 29487 leader_election.cc:290] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:00.971730 29491 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:00.971784 29487 sys_catalog.cc:565] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:00.971813 29491 raft_consensus.cc:697] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [term 1 LEADER]: Becoming Leader. State: Replica: a7841e50f69544b08c06deca9b5c2e47, State: Running, Role: LEADER
I20260201 03:08:00.971932 29491 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } }
I20260201 03:08:00.972352 29492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7841e50f69544b08c06deca9b5c2e47. Latest consensus state: current_term: 1 leader_uuid: "a7841e50f69544b08c06deca9b5c2e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } } }
I20260201 03:08:00.972412 29490 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7841e50f69544b08c06deca9b5c2e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7841e50f69544b08c06deca9b5c2e47" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 40669 } } }
I20260201 03:08:00.972514 29492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:00.972728 29490 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:00.972870 29499 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:00.972994 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 29419
I20260201 03:08:00.973101 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:00.973487 29499 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:00.973758 29396 external_mini_cluster.cc:1468] Setting key 3f0907024ee3fb5de4caa69e7f067987
I20260201 03:08:00.975067 29499 catalog_manager.cc:1357] Generated new cluster ID: 033b86b7602846b2a9684e4f80da0678
I20260201 03:08:00.975117 29499 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:08:00 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:00.982069 29499 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:00.982592 29499 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:00.985656 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:00.974655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.28.181.62:40669)
0201 03:08:00.974845 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:00.974855 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:00.974879 (+ 24us) server_negotiation.cc:408] Connection header received
0201 03:08:00.976021 (+ 1142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:00.976026 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:00.976064 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:00.976176 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:00.977421 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:00.978446 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:00.979078 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:00.979280 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:00.982021 (+ 2741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:00.982025 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:00.982025 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:00.982040 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:00.984289 (+ 2249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:00.984657 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:00.984660 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:00.984660 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:00.984709 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:00.984890 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:00.984891 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:00.984891 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:00.985079 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:00.985200 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:00.985401 (+ 201us) server_negotiation.cc:300] Negotiation successful
0201 03:08:00.985510 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":61,"threads_started":1}
I20260201 03:08:00.986545 29499 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a7841e50f69544b08c06deca9b5c2e47: Generated new TSK 0
I20260201 03:08:00.986668 29499 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:01.026325 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:40669
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:01.092229 29515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:01.092396 29515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:01.092427 29515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:01.093657 29515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:01.093742 29515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:01.093799 29515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:01.095227 29515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:40669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:01.095463 29515 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:01.095641 29515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:01.097694 29521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:01.097685 29523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:01.097671 29520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:01.098151 29515 server_base.cc:1060] running on GCE node
I20260201 03:08:01.098286 29515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:01.098470 29515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:01.099697 29515 hybrid_clock.cc:648] HybridClock initialized: now 1769915281099682 us; error 30 us; skew 500 ppm
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:01.101732 29515 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:01.102246 29515 webserver.cc:492] Webserver started at http://127.28.181.1:37115/ using document root <none> and password file <none>
I20260201 03:08:01.102437 29515 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:01.102484 29515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:01.102613 29515 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:01.103402 29515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "a2581f5bdb7f4d3194c552a6c46dbbe7"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "582ff4d2fc4033d92ce464067b07d9f0"
server_key_iv: "3de11cbe88f7a2846553ea8334193200"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.103693 29515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "a2581f5bdb7f4d3194c552a6c46dbbe7"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "582ff4d2fc4033d92ce464067b07d9f0"
server_key_iv: "3de11cbe88f7a2846553ea8334193200"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.104692 29515 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:01.105261 29530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.105469 29515 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.105552 29515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "a2581f5bdb7f4d3194c552a6c46dbbe7"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "582ff4d2fc4033d92ce464067b07d9f0"
server_key_iv: "3de11cbe88f7a2846553ea8334193200"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.105600 29515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:01.113169 29515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:01.114499 29515 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:01.114578 29515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:01.114717 29515 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:01.114900 29515 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:01.114924 29515 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.114940 29515 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:01.114951 29515 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.119728 29515 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:42155
I20260201 03:08:01.119810 29643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:42155 every 8 connection(s)
I20260201 03:08:01.120029 29515 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:01.126312 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.120564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:36679 (local address 127.28.181.62:40669)
0201 03:08:01.120644 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:01.120646 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:01.120729 (+ 83us) server_negotiation.cc:408] Connection header received
0201 03:08:01.121078 (+ 349us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:01.121079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:01.121087 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:01.121135 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:01.121669 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.122120 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.122552 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.122651 (+ 99us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.124436 (+ 1785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:01.124439 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:01.124440 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:01.124445 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:01.125431 (+ 986us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.125703 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.125703 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.125704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.125720 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.125904 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.125904 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.125904 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.125970 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:01.126012 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:01.126192 (+ 180us) server_negotiation.cc:300] Negotiation successful
0201 03:08:01.126256 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:08:01.126379 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.120517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:40669 (local address 127.28.181.1:36679)
0201 03:08:01.120629 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:01.120655 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:01.121014 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:01.121169 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:01.121172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:01.121386 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:01.121608 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.121610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.122185 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.122185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:01.122494 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.122495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.122576 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.123268 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:01.123281 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:01.124370 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:01.125490 (+ 1120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.125494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.125496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.125647 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.125803 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.125806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.125806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.125866 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.126015 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:01.126018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:01.126097 (+ 79us) client_negotiation.cc:769] Sending connection context
0201 03:08:01.126164 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:08:01.126241 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":48,"threads_started":1}
I20260201 03:08:01.126734 29644 heartbeater.cc:344] Connected to a master server at 127.28.181.62:40669
I20260201 03:08:01.126820 29644 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:01.126987 29644 heartbeater.cc:507] Master 127.28.181.62:40669 requested a full tablet report, sending...
I20260201 03:08:01.127393 29451 ts_manager.cc:194] Registered new tserver with Master: a2581f5bdb7f4d3194c552a6c46dbbe7 (127.28.181.1:42155)
I20260201 03:08:01.127904 29451 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:36679
I20260201 03:08:01.129331 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 29515
I20260201 03:08:01.129390 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:01.129484 29396 external_mini_cluster.cc:1468] Setting key 7205def8d66a19f306ce4e2c512df3da
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:01.163858 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:40669
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:01.228909 29651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:01.229066 29651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:01.229095 29651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:01.230285 29651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:01.230343 29651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:01.230379 29651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:01.231663 29651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:40669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:01.231856 29651 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:01.232015 29651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:01.234093 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:01.234105 29651 server_base.cc:1060] running on GCE node
W20260201 03:08:01.234112 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:01.234112 29656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:01.234390 29651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:01.234611 29651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:01.235738 29651 hybrid_clock.cc:648] HybridClock initialized: now 1769915281235726 us; error 30 us; skew 500 ppm
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:01.237617 29651 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:01.238124 29651 webserver.cc:492] Webserver started at http://127.28.181.2:42693/ using document root <none> and password file <none>
I20260201 03:08:01.238294 29651 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:01.238325 29651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:01.238427 29651 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:01.239171 29651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "c8dd262492624893b949d44592765efd"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "12fe8ff7ff3ee285d70c630397899179"
server_key_iv: "96a3dcf90e8a9f11468501eb66cfc53e"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.239424 29651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "c8dd262492624893b949d44592765efd"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "12fe8ff7ff3ee285d70c630397899179"
server_key_iv: "96a3dcf90e8a9f11468501eb66cfc53e"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.240366 29651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:01.240898 29666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.241046 29651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:01.241117 29651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "c8dd262492624893b949d44592765efd"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "12fe8ff7ff3ee285d70c630397899179"
server_key_iv: "96a3dcf90e8a9f11468501eb66cfc53e"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.241179 29651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:01.257974 29651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:01.259202 29651 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:01.259317 29651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:01.259485 29651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:01.259732 29651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:01.259780 29651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.259814 29651 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:01.259848 29651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.264622 29651 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:37031
I20260201 03:08:01.264662 29779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:37031 every 8 connection(s)
I20260201 03:08:01.264873 29651 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:01.266743 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 29651
I20260201 03:08:01.266810 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:01.266942 29396 external_mini_cluster.cc:1468] Setting key 38d4a5ddd514c8affd264929bda3bb53
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:01.272006 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.265390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:42825 (local address 127.28.181.62:40669)
0201 03:08:01.265444 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:01.265446 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:01.265584 (+ 138us) server_negotiation.cc:408] Connection header received
0201 03:08:01.265927 (+ 343us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:01.265928 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:01.265936 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:01.265985 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:01.266504 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.266988 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.267596 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.267767 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.269656 (+ 1889us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:01.269659 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:01.269660 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:01.269666 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:01.270933 (+ 1267us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.271270 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.271271 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.271271 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.271300 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.271479 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.271480 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.271480 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.271562 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:01.271633 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:01.271848 (+ 215us) server_negotiation.cc:300] Negotiation successful
0201 03:08:01.271923 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:08:01.272068 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.265372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:40669 (local address 127.28.181.2:42825)
0201 03:08:01.265480 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:01.265503 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:01.265866 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:01.266020 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:01.266023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:01.266233 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:01.266447 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.266449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.267083 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.267084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:01.267522 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.267524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.267647 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.268389 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:01.268402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:01.269578 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:01.271020 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.271025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.271027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.271195 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.271353 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.271354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.271354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.271421 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.271626 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:01.271631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:01.271730 (+ 99us) client_negotiation.cc:769] Sending connection context
0201 03:08:01.271822 (+ 92us) client_negotiation.cc:240] Negotiation successful
0201 03:08:01.271923 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":62,"threads_started":1}
I20260201 03:08:01.272377 29780 heartbeater.cc:344] Connected to a master server at 127.28.181.62:40669
I20260201 03:08:01.272451 29780 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:01.272603 29780 heartbeater.cc:507] Master 127.28.181.62:40669 requested a full tablet report, sending...
I20260201 03:08:01.272941 29451 ts_manager.cc:194] Registered new tserver with Master: c8dd262492624893b949d44592765efd (127.28.181.2:37031)
I20260201 03:08:01.273317 29451 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:42825
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:01.302050 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:40669
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:01.367449 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:01.367584 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:01.367604 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:01.368775 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:01.368811 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:01.368832 29787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:01.370136 29787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:40669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:01.370270 29787 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:01.370438 29787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:01.372453 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:01.372459 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:01.372449 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:01.372802 29787 server_base.cc:1060] running on GCE node
I20260201 03:08:01.372936 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:01.373090 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:01.374223 29787 hybrid_clock.cc:648] HybridClock initialized: now 1769915281374207 us; error 31 us; skew 500 ppm
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:01.376277 29787 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:01.376735 29787 webserver.cc:492] Webserver started at http://127.28.181.3:34961/ using document root <none> and password file <none>
I20260201 03:08:01.376897 29787 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:01.376942 29787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:01.377044 29787 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:01.377856 29787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "5e925a4ee4e74910a992f4c999d9822d"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "0250c149423143e67330403a6ec21124"
server_key_iv: "5b7a3c46884fe7d22fe4d911583c40bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.378144 29787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "5e925a4ee4e74910a992f4c999d9822d"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "0250c149423143e67330403a6ec21124"
server_key_iv: "5b7a3c46884fe7d22fe4d911583c40bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.379132 29787 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:01.379676 29802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.379814 29787 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:01.379884 29787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "5e925a4ee4e74910a992f4c999d9822d"
format_stamp: "Formatted at 2026-02-01 03:08:01 on dist-test-slave-ck1n"
server_key: "0250c149423143e67330403a6ec21124"
server_key_iv: "5b7a3c46884fe7d22fe4d911583c40bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:01.379945 29787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:01.396508 29787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:01.398115 29787 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:01.398236 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:01.398411 29787 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:01.398648 29787 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:01.398692 29787 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.398720 29787 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:01.398742 29787 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:01.403787 29787 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:32927
I20260201 03:08:01.403820 29915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:32927 every 8 connection(s)
I20260201 03:08:01.404069 29787 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:01.405040 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 29787
I20260201 03:08:01.405124 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:01.405303 29396 external_mini_cluster.cc:1468] Setting key 287aeb63681b69cc591a6a1044e83b0e
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:01.410856 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.404630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:32821 (local address 127.28.181.62:40669)
0201 03:08:01.404695 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:01.404697 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:01.404794 (+ 97us) server_negotiation.cc:408] Connection header received
0201 03:08:01.405248 (+ 454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:01.405249 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:01.405263 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:01.405327 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:01.406164 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.406584 (+ 420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.407012 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.407106 (+ 94us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.408883 (+ 1777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:01.408886 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:01.408887 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:01.408892 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:01.409956 (+ 1064us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.410229 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.410230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.410230 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.410265 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.410413 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.410414 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.410414 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.410482 (+ 68us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:01.410532 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:01.410738 (+ 206us) server_negotiation.cc:300] Negotiation successful
0201 03:08:01.410801 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:08:01.410910 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.404561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:40669 (local address 127.28.181.3:32821)
0201 03:08:01.404697 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:01.404728 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:01.405106 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:01.405387 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:01.405391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:01.405735 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:01.406086 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.406090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.406645 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.406646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:01.406955 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:01.406956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.407042 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.407658 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:01.407669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:01.408797 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:01.410014 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.410018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.410020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.410175 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.410316 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:01.410316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:01.410316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:01.410372 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:01.410534 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:01.410537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:01.410655 (+ 118us) client_negotiation.cc:769] Sending connection context
0201 03:08:01.410722 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:08:01.410799 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260201 03:08:01.411320 29916 heartbeater.cc:344] Connected to a master server at 127.28.181.62:40669
I20260201 03:08:01.411379 29916 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:01.411540 29916 heartbeater.cc:507] Master 127.28.181.62:40669 requested a full tablet report, sending...
I20260201 03:08:01.411840 29451 ts_manager.cc:194] Registered new tserver with Master: 5e925a4ee4e74910a992f4c999d9822d (127.28.181.3:32927)
I20260201 03:08:01.412122 29451 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:32821
I20260201 03:08:01.416895 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:01.434196 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.428538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.28.181.1:42155)
0201 03:08:01.428699 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:01.428704 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:01.428711 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:01.428730 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:01.428732 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:01.428746 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:01.428797 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:01.429315 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.429938 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.430379 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.430495 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.432141 (+ 1646us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:01.432148 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:01.432149 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:01.432154 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:01.433327 (+ 1173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.433601 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.433603 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.433604 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.433629 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.433749 (+ 120us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.433750 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.433750 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.433861 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:01.433957 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:01.434015 (+ 58us) server_negotiation.cc:300] Negotiation successful
0201 03:08:01.434071 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260201 03:08:01.434496 29558 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55744
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:01.450568 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:01.445498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.28.181.1:42155)
0201 03:08:01.445558 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:01.445560 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:01.445618 (+ 58us) server_negotiation.cc:408] Connection header received
0201 03:08:01.445691 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:01.445692 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:01.445704 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:01.445764 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:01.446299 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.446684 (+ 385us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:01.447170 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:01.447335 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:01.448761 (+ 1426us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:01.448764 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:01.448765 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:01.448769 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:01.449946 (+ 1177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.450165 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.450165 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.450165 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.450180 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:01.450275 (+ 95us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:01.450275 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:01.450275 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:01.450321 (+ 46us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:01.450374 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:01.450443 (+ 69us) server_negotiation.cc:300] Negotiation successful
0201 03:08:01.450518 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:08:01.450984 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 29515
I20260201 03:08:01.455801 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 29651
I20260201 03:08:01.460376 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 29787
I20260201 03:08:01.464206 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 29419
2026-02-01T03:08:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2419 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29930](info): set up 2 sockets
Feb 01 03:08:01 dist-test-slave-ck1n krb5kdc[29930](info): commencing operation
krb5kdc: starting...
W20260201 03:08:03.191823 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.709s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:03.297685 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:41799
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:03.363649 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:03.363809 29946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:03.363827 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:03.365070 29946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:03.365106 29946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:03.365119 29946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:03.365129 29946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:03.365139 29946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:03.366520 29946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:41799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:03.366712 29946 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:03.366863 29946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:03.368772 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.368858 29946 server_base.cc:1060] running on GCE node
W20260201 03:08:03.368765 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.368765 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.369164 29946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:03.369366 29946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:03.370481 29946 hybrid_clock.cc:648] HybridClock initialized: now 1769915283370467 us; error 29 us; skew 500 ppm
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:03.372238 29946 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:03.372608 29946 webserver.cc:492] Webserver started at http://127.28.181.62:32971/ using document root <none> and password file <none>
I20260201 03:08:03.372725 29946 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:03.372751 29946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:03.372820 29946 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:03.373574 29946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "19a49e401fbd42798cdb081e45b7a85b"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "133693dfdb397f476aebb0c29b2e6243"
server_key_iv: "823419221dc12057cc7f8bca16da1bfb"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.373858 29946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "19a49e401fbd42798cdb081e45b7a85b"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "133693dfdb397f476aebb0c29b2e6243"
server_key_iv: "823419221dc12057cc7f8bca16da1bfb"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.374805 29946 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:03.375314 29961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.375458 29946 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:03.375531 29946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "19a49e401fbd42798cdb081e45b7a85b"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "133693dfdb397f476aebb0c29b2e6243"
server_key_iv: "823419221dc12057cc7f8bca16da1bfb"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.375594 29946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:03.392822 29946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:03.398015 29946 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:03.398129 29946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:03.401302 29946 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:41799
I20260201 03:08:03.401348 30013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:41799 every 8 connection(s)
I20260201 03:08:03.401636 29946 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:03.402209 30014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:03.404318 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b: Bootstrap starting.
I20260201 03:08:03.404779 30014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:03.405002 30014 log.cc:826] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:03.405557 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b: No bootstrap required, opened a new log
I20260201 03:08:03.406518 30014 raft_consensus.cc:359] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } }
I20260201 03:08:03.406612 30014 raft_consensus.cc:385] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:03.406628 30014 raft_consensus.cc:740] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a49e401fbd42798cdb081e45b7a85b, State: Initialized, Role: FOLLOWER
I20260201 03:08:03.406734 30014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } }
I20260201 03:08:03.406793 30014 raft_consensus.cc:399] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:03.406829 30014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:03.406873 30014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:03.407310 30014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } }
I20260201 03:08:03.407421 30014 leader_election.cc:304] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a49e401fbd42798cdb081e45b7a85b; no voters:
I20260201 03:08:03.407583 30014 leader_election.cc:290] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:03.407624 30017 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:03.407737 30017 raft_consensus.cc:697] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [term 1 LEADER]: Becoming Leader. State: Replica: 19a49e401fbd42798cdb081e45b7a85b, State: Running, Role: LEADER
I20260201 03:08:03.407788 30014 sys_catalog.cc:565] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:03.407809 30017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } }
I20260201 03:08:03.408136 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a49e401fbd42798cdb081e45b7a85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } } }
I20260201 03:08:03.408156 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a49e401fbd42798cdb081e45b7a85b. Latest consensus state: current_term: 1 leader_uuid: "19a49e401fbd42798cdb081e45b7a85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a49e401fbd42798cdb081e45b7a85b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 41799 } } }
I20260201 03:08:03.408267 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:03.408275 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:03.408593 30027 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:03.409148 30027 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:03.410339 30027 catalog_manager.cc:1357] Generated new cluster ID: 634541af79a44a5ab20eb2a8a4f26084
I20260201 03:08:03.410404 30027 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:03.410928 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 29946
I20260201 03:08:03.411006 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:03.411144 29396 external_mini_cluster.cc:1468] Setting key 391cb9f5f113556d40c19ae8b1044869
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:03.420275 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.411499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.28.181.62:41799)
0201 03:08:03.411625 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.411638 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.411769 (+ 131us) server_negotiation.cc:408] Connection header received
0201 03:08:03.411928 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.411933 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.411971 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.412045 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.412945 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.414003 (+ 1058us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.414856 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.415061 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.417119 (+ 2058us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.417125 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.417125 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.417142 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.418766 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.419072 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.419074 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.419074 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.419131 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.419271 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.419271 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.419272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.419453 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.419569 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.419935 (+ 366us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.420032 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260201 03:08:03.429728 30027 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:03.430145 30027 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:03.434219 30027 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 19a49e401fbd42798cdb081e45b7a85b: Generated new TSK 0
I20260201 03:08:03.434360 30027 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:03.481878 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:41799
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:03.564056 30042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:03.564204 30042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:03.564234 30042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:03.565469 30042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:03.565528 30042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:03.565567 30042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:03.566947 30042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:41799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:03.567147 30042 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:03.567307 30042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:03.569432 30047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.569454 30050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.569430 30048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.569821 30042 server_base.cc:1060] running on GCE node
I20260201 03:08:03.569981 30042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:03.570151 30042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:03.571282 30042 hybrid_clock.cc:648] HybridClock initialized: now 1769915283571268 us; error 31 us; skew 500 ppm
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:03.573212 30042 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:03.573709 30042 webserver.cc:492] Webserver started at http://127.28.181.1:39615/ using document root <none> and password file <none>
I20260201 03:08:03.573907 30042 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:03.573962 30042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:03.574083 30042 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:03.574884 30042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "8a43cb49c58442909ab9279c64bd1ac2"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "9b9c003494142ddef3f4df4e8d22199c"
server_key_iv: "fb7c51dfc8c4d93be2fff8544343918b"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.575167 30042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "8a43cb49c58442909ab9279c64bd1ac2"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "9b9c003494142ddef3f4df4e8d22199c"
server_key_iv: "fb7c51dfc8c4d93be2fff8544343918b"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.576156 30042 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:03.576699 30057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.576839 30042 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:03.576920 30042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "8a43cb49c58442909ab9279c64bd1ac2"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "9b9c003494142ddef3f4df4e8d22199c"
server_key_iv: "fb7c51dfc8c4d93be2fff8544343918b"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.576987 30042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:03.585637 30042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:03.586900 30042 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:03.587023 30042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:03.587203 30042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:03.587456 30042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:03.587504 30042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.587529 30042 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:03.587555 30042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.592342 30042 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:45457
I20260201 03:08:03.592374 30170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:45457 every 8 connection(s)
I20260201 03:08:03.592602 30042 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:03.595207 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30042
I20260201 03:08:03.595294 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:03.595443 29396 external_mini_cluster.cc:1468] Setting key b1b62a1ebe3e07f4d9def564a70833b6
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:03.599731 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.593126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:41799 (local address 127.28.181.1:35191)
0201 03:08:03.593238 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.593265 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.593623 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.593839 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.593842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.594076 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:03.594295 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.594298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.594832 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.594833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.595231 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.595233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.595627 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.596051 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:03.596071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:03.597314 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:03.598837 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.598842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.598843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.598996 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.599144 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.599145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.599145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.599186 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.599385 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:03.599387 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:03.599468 (+ 81us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.599552 (+ 84us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.599631 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
I20260201 03:08:03.599821 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.593174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:35191 (local address 127.28.181.62:41799)
0201 03:08:03.593227 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.593229 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.593336 (+ 107us) server_negotiation.cc:408] Connection header received
0201 03:08:03.593687 (+ 351us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.593688 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.593698 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.593746 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.594361 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.594770 (+ 409us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.595306 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.595421 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.597392 (+ 1971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.597396 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.597396 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.597403 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.598768 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.599062 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.599063 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.599064 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.599088 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.599236 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.599236 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.599236 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.599332 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.599394 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.599606 (+ 212us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.599700 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260201 03:08:03.600286 30171 heartbeater.cc:344] Connected to a master server at 127.28.181.62:41799
I20260201 03:08:03.600382 30171 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:03.600558 30171 heartbeater.cc:507] Master 127.28.181.62:41799 requested a full tablet report, sending...
I20260201 03:08:03.600965 29978 ts_manager.cc:194] Registered new tserver with Master: 8a43cb49c58442909ab9279c64bd1ac2 (127.28.181.1:45457)
I20260201 03:08:03.601482 29978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:35191
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:03.632272 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:41799
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:03.697218 30178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:03.697341 30178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:03.697360 30178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:03.698607 30178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:03.698647 30178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:03.698669 30178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:03.699991 30178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:41799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:03.700129 30178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:03.700292 30178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:03.702298 30186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.702296 30183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.702368 30184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.702530 30178 server_base.cc:1060] running on GCE node
I20260201 03:08:03.702673 30178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:03.702858 30178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:03.703974 30178 hybrid_clock.cc:648] HybridClock initialized: now 1769915283703951 us; error 37 us; skew 500 ppm
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:03.705950 30178 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:03.706409 30178 webserver.cc:492] Webserver started at http://127.28.181.2:39333/ using document root <none> and password file <none>
I20260201 03:08:03.706569 30178 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:03.706601 30178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:03.706707 30178 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:03.707443 30178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "754d4264a6824f998e468613c18ecacd"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "64d215eed214a95f0166bbbd1e2230bb"
server_key_iv: "1c83d48c99660452f34eb0f3a6070d29"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.707713 30178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "754d4264a6824f998e468613c18ecacd"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "64d215eed214a95f0166bbbd1e2230bb"
server_key_iv: "1c83d48c99660452f34eb0f3a6070d29"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.709030 30178 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:03.709637 30193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.709805 30178 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:03.709867 30178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "754d4264a6824f998e468613c18ecacd"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "64d215eed214a95f0166bbbd1e2230bb"
server_key_iv: "1c83d48c99660452f34eb0f3a6070d29"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.709906 30178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:03.731580 30178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:03.733202 30178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:03.733287 30178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:03.733428 30178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:03.733704 30178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:03.733734 30178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.733752 30178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:03.733817 30178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.738435 30178 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:36965
I20260201 03:08:03.738497 30306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:36965 every 8 connection(s)
I20260201 03:08:03.738734 30178 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:03.744951 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.739284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:32875 (local address 127.28.181.62:41799)
0201 03:08:03.739348 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.739350 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.739456 (+ 106us) server_negotiation.cc:408] Connection header received
0201 03:08:03.739837 (+ 381us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.739838 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.739851 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.739892 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.740451 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.740859 (+ 408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.741283 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.741373 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.743050 (+ 1677us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.743054 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.743054 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.743060 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.744117 (+ 1057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.744390 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.744390 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.744391 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.744407 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.744537 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.744537 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.744537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.744604 (+ 67us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.744645 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.744833 (+ 188us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.744894 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:03.745108 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.739247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:41799 (local address 127.28.181.2:32875)
0201 03:08:03.739371 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.739393 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.739766 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.739933 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.739936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.740162 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:03.740392 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.740394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.740920 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.740921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.741229 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.741230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.741500 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.741863 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:03.741876 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:03.742976 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:03.744177 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.744182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.744183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.744336 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.744450 (+ 114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.744450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.744450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.744492 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.744649 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:03.744652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:03.744736 (+ 84us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.744824 (+ 88us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.744903 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260201 03:08:03.745451 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30178
I20260201 03:08:03.745589 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:03.745571 30307 heartbeater.cc:344] Connected to a master server at 127.28.181.62:41799
I20260201 03:08:03.745673 30307 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:03.745730 29396 external_mini_cluster.cc:1468] Setting key 4ef83fc4f83e83752b4c919734081a91
I20260201 03:08:03.745905 30307 heartbeater.cc:507] Master 127.28.181.62:41799 requested a full tablet report, sending...
I20260201 03:08:03.746206 29977 ts_manager.cc:194] Registered new tserver with Master: 754d4264a6824f998e468613c18ecacd (127.28.181.2:36965)
I20260201 03:08:03.746654 29977 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:32875
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:03.780364 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:41799
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:03.845258 30314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:03.845409 30314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:03.845441 30314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:03.846671 30314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:03.846731 30314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:03.846768 30314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:03.848079 30314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:41799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:03.848275 30314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:03.848440 30314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:03.850422 30319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:03.850468 30320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.850476 30314 server_base.cc:1060] running on GCE node
W20260201 03:08:03.850452 30322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:03.850734 30314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:03.850870 30314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:03.851986 30314 hybrid_clock.cc:648] HybridClock initialized: now 1769915283851977 us; error 25 us; skew 500 ppm
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:03.854041 30314 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:03.854512 30314 webserver.cc:492] Webserver started at http://127.28.181.3:34225/ using document root <none> and password file <none>
I20260201 03:08:03.854691 30314 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:03.854737 30314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:03.854841 30314 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:03.855582 30314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "9b168074d9c4482583f3a4b67e9cc6cb"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "5600bd3ee590e6ee18157cea6123b5b2"
server_key_iv: "c97a3cb1db7bf6a164584db07a126e56"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.855863 30314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "9b168074d9c4482583f3a4b67e9cc6cb"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "5600bd3ee590e6ee18157cea6123b5b2"
server_key_iv: "c97a3cb1db7bf6a164584db07a126e56"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.856840 30314 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:03.857375 30329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.857517 30314 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:03.857594 30314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "9b168074d9c4482583f3a4b67e9cc6cb"
format_stamp: "Formatted at 2026-02-01 03:08:03 on dist-test-slave-ck1n"
server_key: "5600bd3ee590e6ee18157cea6123b5b2"
server_key_iv: "c97a3cb1db7bf6a164584db07a126e56"
server_key_version: "encryptionkey@0"
I20260201 03:08:03.857664 30314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:03.868459 30314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:03.869721 30314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:03.869867 30314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:03.870043 30314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:03.870292 30314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:03.870338 30314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.870373 30314 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:03.870398 30314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:03.875602 30314 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:34109
I20260201 03:08:03.875635 30442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:34109 every 8 connection(s)
I20260201 03:08:03.875866 30314 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:03.882368 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.876440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:52775 (local address 127.28.181.62:41799)
0201 03:08:03.876502 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.876505 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.876622 (+ 117us) server_negotiation.cc:408] Connection header received
0201 03:08:03.877020 (+ 398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.877021 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.877030 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.877084 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.877709 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.878173 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.878677 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.878774 (+ 97us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.880545 (+ 1771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.880548 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.880549 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.880553 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.881508 (+ 955us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.881816 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.881816 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.881817 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.881835 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.881974 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.881974 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.881974 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.882037 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.882074 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.882253 (+ 179us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.882313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:03.882411 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.876389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:41799 (local address 127.28.181.3:52775)
0201 03:08:03.876517 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.876545 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.876952 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.877140 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.877143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.877390 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:03.877648 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.877649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.878235 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.878235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.878609 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.878610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.878901 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.879277 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:03.879290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:03.880483 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:03.881567 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.881571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.881573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.881723 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.881889 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:03.881890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:03.881890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:03.881933 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:03.882075 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:03.882077 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:03.882155 (+ 78us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.882233 (+ 78us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.882310 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20260201 03:08:03.882800 30443 heartbeater.cc:344] Connected to a master server at 127.28.181.62:41799
I20260201 03:08:03.882866 30443 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:03.882997 30443 heartbeater.cc:507] Master 127.28.181.62:41799 requested a full tablet report, sending...
I20260201 03:08:03.883262 29977 ts_manager.cc:194] Registered new tserver with Master: 9b168074d9c4482583f3a4b67e9cc6cb (127.28.181.3:34109)
I20260201 03:08:03.883533 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30314
I20260201 03:08:03.883620 29977 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:52775
I20260201 03:08:03.883642 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:03.883797 29396 external_mini_cluster.cc:1468] Setting key 7c2a9714cfbaccc4323f56c04b099f98
I20260201 03:08:03.884608 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:08:03.890014 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.885814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.28.181.62:41799)
0201 03:08:03.885867 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.885869 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.885925 (+ 56us) server_negotiation.cc:408] Connection header received
0201 03:08:03.885987 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.885988 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.885996 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.886057 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.886598 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.886969 (+ 371us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.887472 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.887629 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.888266 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.888269 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.888269 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.888275 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.889155 (+ 880us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.889428 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.889428 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.889429 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.889451 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.889580 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.889580 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.889580 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.889662 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.889757 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.889871 (+ 114us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.889932 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:08:03.891482 29977 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:03.892570 29977 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:03.896771 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:45457 (local address 127.0.0.1:46598)
0201 03:08:03.894652 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.894660 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.894695 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.894871 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.894873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.894876 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:03.895086 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.895088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.895950 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.895951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.896559 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.896561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896616 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.896646 (+ 30us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.896683 (+ 37us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.896714 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":54,"threads_started":1}
I20260201 03:08:03.897066 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:34109 (local address 127.0.0.1:53656)
0201 03:08:03.894509 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.894526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.894574 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.895082 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.895082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.895084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:03.895249 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.895250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896365 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.896366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.896906 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.896907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896962 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.896965 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.896992 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.897021 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":48,"threads_started":1}
I20260201 03:08:03.897272 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36965 (local address 127.0.0.1:48758)
0201 03:08:03.894764 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.894772 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.894821 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.894980 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.894981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.894983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:03.895163 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.895165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896357 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.896358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.897088 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.897090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.897146 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.897148 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.897176 (+ 28us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.897207 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":7,"thread_start_us":57,"threads_started":1}
I20260201 03:08:03.897712 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.28.181.1:45457)
0201 03:08:03.894685 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.894693 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.894702 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:03.894757 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.894758 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.894774 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.894837 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:03.895146 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.895867 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.896967 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.897545 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.897584 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.897610 (+ 26us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.897644 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260201 03:08:03.897725 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.28.181.3:34109)
0201 03:08:03.894906 (+ 574us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.894912 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.894918 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:03.894963 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.894964 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.894977 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.895060 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:03.895321 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896271 (+ 950us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.897230 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.897622 (+ 392us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.897647 (+ 25us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.897666 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.897687 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":54,"threads_started":1}
I20260201 03:08:03.898027 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.894523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.28.181.2:36965)
0201 03:08:03.894669 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.894675 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.894847 (+ 172us) server_negotiation.cc:408] Connection header received
0201 03:08:03.894882 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.894884 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.894901 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.894971 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:03.895289 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.896272 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.897406 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.897881 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.897917 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.897942 (+ 25us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.897978 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260201 03:08:03.898171 30377 tablet_service.cc:1505] Processing CreateTablet for tablet 7e98824c31934ca0accf059e9f4c6ba0 (DEFAULT_TABLE table=test-table [id=81fed03946dd4e099884833152b40d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:03.898283 30105 tablet_service.cc:1505] Processing CreateTablet for tablet 7e98824c31934ca0accf059e9f4c6ba0 (DEFAULT_TABLE table=test-table [id=81fed03946dd4e099884833152b40d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:03.898413 30377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e98824c31934ca0accf059e9f4c6ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:03.898494 30105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e98824c31934ca0accf059e9f4c6ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:03.898521 30241 tablet_service.cc:1505] Processing CreateTablet for tablet 7e98824c31934ca0accf059e9f4c6ba0 (DEFAULT_TABLE table=test-table [id=81fed03946dd4e099884833152b40d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:03.898783 30241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e98824c31934ca0accf059e9f4c6ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:03.900483 30457 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Bootstrap starting.
I20260201 03:08:03.900956 30459 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: Bootstrap starting.
I20260201 03:08:03.901116 30457 tablet_bootstrap.cc:654] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:03.901345 30457 log.cc:826] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:03.901463 30459 tablet_bootstrap.cc:654] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:03.901705 30459 log.cc:826] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:03.901948 30457 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: No bootstrap required, opened a new log
I20260201 03:08:03.901988 30458 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: Bootstrap starting.
I20260201 03:08:03.902016 30457 ts_tablet_manager.cc:1403] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:03.902316 30459 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: No bootstrap required, opened a new log
I20260201 03:08:03.902374 30459 ts_tablet_manager.cc:1403] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:03.902460 30458 tablet_bootstrap.cc:654] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:03.902707 30458 log.cc:826] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:03.903311 30458 tablet_bootstrap.cc:492] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: No bootstrap required, opened a new log
I20260201 03:08:03.903368 30458 ts_tablet_manager.cc:1403] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:03.903328 30457 raft_consensus.cc:359] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.903477 30457 raft_consensus.cc:385] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:03.903512 30457 raft_consensus.cc:740] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754d4264a6824f998e468613c18ecacd, State: Initialized, Role: FOLLOWER
I20260201 03:08:03.903590 30457 consensus_queue.cc:260] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.903833 30457 ts_tablet_manager.cc:1434] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:03.903766 30459 raft_consensus.cc:359] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.903883 30459 raft_consensus.cc:385] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:03.903909 30459 raft_consensus.cc:740] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a43cb49c58442909ab9279c64bd1ac2, State: Initialized, Role: FOLLOWER
I20260201 03:08:03.903949 30307 heartbeater.cc:499] Master 127.28.181.62:41799 was elected leader, sending a full tablet report...
I20260201 03:08:03.903986 30459 consensus_queue.cc:260] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.904333 30459 ts_tablet_manager.cc:1434] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:03.904434 30171 heartbeater.cc:499] Master 127.28.181.62:41799 was elected leader, sending a full tablet report...
I20260201 03:08:03.904901 30458 raft_consensus.cc:359] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.905020 30458 raft_consensus.cc:385] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:03.905055 30458 raft_consensus.cc:740] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b168074d9c4482583f3a4b67e9cc6cb, State: Initialized, Role: FOLLOWER
I20260201 03:08:03.905148 30458 consensus_queue.cc:260] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.905344 30458 ts_tablet_manager.cc:1434] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:03.905388 30443 heartbeater.cc:499] Master 127.28.181.62:41799 was elected leader, sending a full tablet report...
I20260201 03:08:03.950330 30464 raft_consensus.cc:493] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:03.950417 30464 raft_consensus.cc:515] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.950699 30464 leader_election.cc:290] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b168074d9c4482583f3a4b67e9cc6cb (127.28.181.3:34109), 754d4264a6824f998e468613c18ecacd (127.28.181.2:36965)
I20260201 03:08:03.953121 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.950886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36965 (local address 127.28.181.1:41743)
0201 03:08:03.951026 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.951035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.951104 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.951214 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.951215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.951216 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:03.951432 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.951433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952185 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.952186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.952882 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.952884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952966 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.952967 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.953000 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.953030 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260201 03:08:03.953502 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.950934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:41743 (local address 127.28.181.2:36965)
0201 03:08:03.950972 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.950974 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.951117 (+ 143us) server_negotiation.cc:408] Connection header received
0201 03:08:03.951140 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.951141 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.951149 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.951192 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:03.951492 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952105 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.952982 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.953417 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.953423 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.953441 (+ 18us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.953461 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:03.953526 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.950918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:38029 (local address 127.28.181.3:34109)
0201 03:08:03.950953 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.950955 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.950962 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:03.951032 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.951033 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.951041 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.951084 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:03.951394 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952033 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.953073 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.953444 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.953450 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.953466 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.953486 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:03.953688 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.950831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:34109 (local address 127.28.181.1:38029)
0201 03:08:03.950920 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:03.950930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:03.950971 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:03.951104 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:03.951105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:03.951107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:03.951328 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.951330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952101 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.952102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:03.952879 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:03.952881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.952963 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.952967 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:03.953000 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:08:03.953030 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260201 03:08:03.953848 30261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e98824c31934ca0accf059e9f4c6ba0" candidate_uuid: "8a43cb49c58442909ab9279c64bd1ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754d4264a6824f998e468613c18ecacd" is_pre_election: true
I20260201 03:08:03.953990 30261 raft_consensus.cc:2468] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a43cb49c58442909ab9279c64bd1ac2 in term 0.
I20260201 03:08:03.953981 30397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e98824c31934ca0accf059e9f4c6ba0" candidate_uuid: "8a43cb49c58442909ab9279c64bd1ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" is_pre_election: true
I20260201 03:08:03.954077 30397 raft_consensus.cc:2468] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a43cb49c58442909ab9279c64bd1ac2 in term 0.
I20260201 03:08:03.954187 30060 leader_election.cc:304] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754d4264a6824f998e468613c18ecacd, 8a43cb49c58442909ab9279c64bd1ac2; no voters:
I20260201 03:08:03.954300 30464 raft_consensus.cc:2804] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:03.954344 30464 raft_consensus.cc:493] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:03.954358 30464 raft_consensus.cc:3060] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:03.954905 30464 raft_consensus.cc:515] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.955021 30464 leader_election.cc:290] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 9b168074d9c4482583f3a4b67e9cc6cb (127.28.181.3:34109), 754d4264a6824f998e468613c18ecacd (127.28.181.2:36965)
I20260201 03:08:03.955132 30397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e98824c31934ca0accf059e9f4c6ba0" candidate_uuid: "8a43cb49c58442909ab9279c64bd1ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b168074d9c4482583f3a4b67e9cc6cb"
I20260201 03:08:03.955170 30261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e98824c31934ca0accf059e9f4c6ba0" candidate_uuid: "8a43cb49c58442909ab9279c64bd1ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754d4264a6824f998e468613c18ecacd"
I20260201 03:08:03.955199 30397 raft_consensus.cc:3060] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:03.955261 30261 raft_consensus.cc:3060] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:03.955832 30397 raft_consensus.cc:2468] T 7e98824c31934ca0accf059e9f4c6ba0 P 9b168074d9c4482583f3a4b67e9cc6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a43cb49c58442909ab9279c64bd1ac2 in term 1.
I20260201 03:08:03.955839 30261 raft_consensus.cc:2468] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a43cb49c58442909ab9279c64bd1ac2 in term 1.
I20260201 03:08:03.955987 30058 leader_election.cc:304] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a43cb49c58442909ab9279c64bd1ac2, 9b168074d9c4482583f3a4b67e9cc6cb; no voters:
I20260201 03:08:03.956084 30464 raft_consensus.cc:2804] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:03.956215 30464 raft_consensus.cc:697] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 8a43cb49c58442909ab9279c64bd1ac2, State: Running, Role: LEADER
I20260201 03:08:03.956324 30464 consensus_queue.cc:237] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } }
I20260201 03:08:03.956965 29978 catalog_manager.cc:5654] T 7e98824c31934ca0accf059e9f4c6ba0 P 8a43cb49c58442909ab9279c64bd1ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a43cb49c58442909ab9279c64bd1ac2 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "8a43cb49c58442909ab9279c64bd1ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b168074d9c4482583f3a4b67e9cc6cb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a43cb49c58442909ab9279c64bd1ac2" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 45457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754d4264a6824f998e468613c18ecacd" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36965 } health_report { overall_health: UNKNOWN } } }
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:03.974550 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.969052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.28.181.1:45457)
0201 03:08:03.969107 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.969109 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.969163 (+ 54us) server_negotiation.cc:408] Connection header received
0201 03:08:03.969196 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.969197 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.969204 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.969244 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.969879 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.970243 (+ 364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.970665 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.970756 (+ 91us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.972458 (+ 1702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.972464 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.972464 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.972469 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.973734 (+ 1265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.974048 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.974050 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.974050 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.974078 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.974206 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.974207 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.974207 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.974272 (+ 65us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.974349 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.974398 (+ 49us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.974451 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260201 03:08:03.989173 30308 tablet.cc:2378] T 7e98824c31934ca0accf059e9f4c6ba0 P 754d4264a6824f998e468613c18ecacd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:03.991946 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:03.987018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.28.181.1:45457)
0201 03:08:03.987075 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:03.987077 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:03.987167 (+ 90us) server_negotiation.cc:408] Connection header received
0201 03:08:03.987195 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:03.987196 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:03.987203 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:03.987237 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:03.987711 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.988070 (+ 359us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:03.988480 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:03.988570 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:03.990325 (+ 1755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:03.990328 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:03.990328 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:03.990332 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:03.991291 (+ 959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.991511 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.991512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.991512 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.991526 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:03.991628 (+ 102us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:03.991628 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:03.991629 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:03.991674 (+ 45us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:03.991724 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:03.991813 (+ 89us) server_negotiation.cc:300] Negotiation successful
0201 03:08:03.991866 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Feb 01 03:08:03 dist-test-slave-ck1n krb5kdc[29930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:08:04 dist-test-slave-ck1n krb5kdc[29930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:04.007985 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:04.003363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.28.181.1:45457)
0201 03:08:04.003413 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:04.003414 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:04.003473 (+ 59us) server_negotiation.cc:408] Connection header received
0201 03:08:04.003506 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:04.003506 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:04.003513 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:04.003558 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:04.004073 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:04.004407 (+ 334us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:04.004818 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:04.004901 (+ 83us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:04.006427 (+ 1526us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:04.006430 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:04.006430 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:04.006435 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:04.007388 (+ 953us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:04.007617 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:04.007618 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:04.007618 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:04.007632 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:04.007727 (+ 95us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:04.007727 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:04.007727 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:04.007773 (+ 46us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:04.007825 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:04.007893 (+ 68us) server_negotiation.cc:300] Negotiation successful
0201 03:08:04.007940 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260201 03:08:04.008307 30085 tablet_service.cc:3012] Rejecting scan request for tablet 7e98824c31934ca0accf059e9f4c6ba0: Uninitialized: clean time has not yet been initialized
I20260201 03:08:04.009014 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30042
I20260201 03:08:04.013362 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30178
I20260201 03:08:04.017335 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30314
I20260201 03:08:04.021669 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 29946
2026-02-01T03:08:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2557 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:04 dist-test-slave-ck1n krb5kdc[30481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:04 dist-test-slave-ck1n krb5kdc[30481](info): set up 2 sockets
Feb 01 03:08:04 dist-test-slave-ck1n krb5kdc[30481](info): commencing operation
krb5kdc: starting...
W20260201 03:08:05.731321 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.690s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:05 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:05.836198 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:43563
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:43563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:05.901362 30497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:05.901494 30497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:05.901510 30497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:05.902639 30497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:05.902673 30497 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:08:05.902684 30497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:05.902695 30497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:05.902705 30497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:05.902715 30497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:05.904029 30497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:43563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:05.904157 30497 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:05.904278 30497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:05.906184 30503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:05.906203 30497 server_base.cc:1060] running on GCE node
W20260201 03:08:05.906198 30502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:05.906318 30505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:05.906481 30497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:05.906644 30497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:05.907781 30497 hybrid_clock.cc:648] HybridClock initialized: now 1769915285907763 us; error 29 us; skew 500 ppm
Feb 01 03:08:05 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915285, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:05.909610 30497 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:05.910074 30497 webserver.cc:492] Webserver started at http://127.28.181.62:41533/ using document root <none> and password file <none>
I20260201 03:08:05.910195 30497 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:05.910221 30497 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:05.910282 30497 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:05.911016 30497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "59993cdfc8ce4be3989cb7320cdeb2a6"
format_stamp: "Formatted at 2026-02-01 03:08:05 on dist-test-slave-ck1n"
server_key: "d6dc059689546cca86b053fd8ad8ca63"
server_key_iv: "69f95312bc749e17f47b028206f3f8b0"
server_key_version: "encryptionkey@0"
I20260201 03:08:05.911244 30497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "59993cdfc8ce4be3989cb7320cdeb2a6"
format_stamp: "Formatted at 2026-02-01 03:08:05 on dist-test-slave-ck1n"
server_key: "d6dc059689546cca86b053fd8ad8ca63"
server_key_iv: "69f95312bc749e17f47b028206f3f8b0"
server_key_version: "encryptionkey@0"
I20260201 03:08:05.912148 30497 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:05.912671 30512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:05.912838 30497 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:05.912901 30497 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "59993cdfc8ce4be3989cb7320cdeb2a6"
format_stamp: "Formatted at 2026-02-01 03:08:05 on dist-test-slave-ck1n"
server_key: "d6dc059689546cca86b053fd8ad8ca63"
server_key_iv: "69f95312bc749e17f47b028206f3f8b0"
server_key_version: "encryptionkey@0"
I20260201 03:08:05.912942 30497 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:05.926345 30497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:05.927598 30497 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:05.927677 30497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:05.931244 30497 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:43563
I20260201 03:08:05.931298 30574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:43563 every 8 connection(s)
I20260201 03:08:05.931563 30497 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:05.932276 30575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:05.933977 30575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Bootstrap starting.
I20260201 03:08:05.934443 30575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:05.934679 30575 log.cc:826] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:05.935303 30575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: No bootstrap required, opened a new log
I20260201 03:08:05.936268 30575 raft_consensus.cc:359] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } }
I20260201 03:08:05.936380 30575 raft_consensus.cc:385] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:05.936410 30575 raft_consensus.cc:740] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59993cdfc8ce4be3989cb7320cdeb2a6, State: Initialized, Role: FOLLOWER
I20260201 03:08:05.936520 30575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } }
I20260201 03:08:05.936587 30575 raft_consensus.cc:399] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:05.936620 30575 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:05.936661 30575 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:05.937078 30575 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } }
I20260201 03:08:05.937181 30575 leader_election.cc:304] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59993cdfc8ce4be3989cb7320cdeb2a6; no voters:
I20260201 03:08:05.937319 30575 leader_election.cc:290] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:05.937358 30578 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:05.937503 30575 sys_catalog.cc:565] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:05.937515 30578 raft_consensus.cc:697] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 59993cdfc8ce4be3989cb7320cdeb2a6, State: Running, Role: LEADER
I20260201 03:08:05.937602 30578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } }
I20260201 03:08:05.937922 30580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59993cdfc8ce4be3989cb7320cdeb2a6. Latest consensus state: current_term: 1 leader_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } } }
I20260201 03:08:05.937918 30579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59993cdfc8ce4be3989cb7320cdeb2a6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43563 } } }
I20260201 03:08:05.937999 30579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:05.938129 30580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:05.938943 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30497
I20260201 03:08:05.939023 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:05.939173 29396 external_mini_cluster.cc:1468] Setting key fcf62fbca37e46e0ac9a79d7a0f2e049
W20260201 03:08:05.939201 30594 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:05.939249 30594 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:05.939287 30585 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:05.939399 30585 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:05.940562 30585 catalog_manager.cc:1357] Generated new cluster ID: 8875b26f272d47cb8c3c7c5c86dbc4fd
I20260201 03:08:05.940606 30585 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:08:05 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:05.945756 30585 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:05.946326 30585 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:05.947156 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:05.939830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.28.181.62:43563)
0201 03:08:05.939972 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:05.939983 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:05.940000 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:08:05.940086 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:05.940090 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:05.940120 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:05.940894 (+ 774us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:05.940957 (+ 63us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:05.941555 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:05.942146 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:05.942354 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:05.944113 (+ 1759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:05.944118 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:05.944120 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:05.944141 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:05.945973 (+ 1832us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:05.946265 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:05.946267 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:05.946267 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:05.946317 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:05.946465 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:05.946465 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:05.946466 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:05.946609 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:05.946743 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:05.946836 (+ 93us) server_negotiation.cc:300] Negotiation successful
0201 03:08:05.947053 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260201 03:08:05.951601 30585 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Generated new TSK 0
I20260201 03:08:05.951745 30585 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:05.986406 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:06.062009 30603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:06.062147 30603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:06.062166 30603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:06.063490 30603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:06.063544 30603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:06.063578 30603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:06.063629 30603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:06.066082 30603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:06.066268 30603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:06.066439 30603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:06.068940 30611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.069396 30608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.069638 30609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:06.069877 30603 server_base.cc:1060] running on GCE node
I20260201 03:08:06.070084 30603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:06.070317 30603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:06.071446 30603 hybrid_clock.cc:648] HybridClock initialized: now 1769915286071431 us; error 32 us; skew 500 ppm
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:06.073518 30603 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:06.074096 30603 webserver.cc:492] Webserver started at http://127.28.181.1:46615/ using document root <none> and password file <none>
I20260201 03:08:06.074271 30603 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:06.074302 30603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:06.074394 30603 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:06.075184 30603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "c7cc69635c5c4e6d93887f01ef211ad3"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "0a9ad1992b18517d7bb52631e271f367"
server_key_iv: "397b69bbf8663038fa9efc42bd57bed4"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.075488 30603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "c7cc69635c5c4e6d93887f01ef211ad3"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "0a9ad1992b18517d7bb52631e271f367"
server_key_iv: "397b69bbf8663038fa9efc42bd57bed4"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.076570 30603 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:06.077203 30618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.077360 30603 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:06.077430 30603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "c7cc69635c5c4e6d93887f01ef211ad3"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "0a9ad1992b18517d7bb52631e271f367"
server_key_iv: "397b69bbf8663038fa9efc42bd57bed4"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.077502 30603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:06.089048 30603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:06.090734 30603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:06.090868 30603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:06.091339 30603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:06.091382 30603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.091410 30603 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:06.091429 30603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:06.098011 30603 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:36007
I20260201 03:08:06.098068 30733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:36007 every 8 connection(s)
I20260201 03:08:06.098320 30603 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:06.098987 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.091581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49353 (local address 127.28.181.62:43563)
0201 03:08:06.091628 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.091630 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.091803 (+ 173us) server_negotiation.cc:408] Connection header received
0201 03:08:06.092217 (+ 414us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.092218 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.092231 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.092276 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.092921 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.093369 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.093964 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.094168 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.096160 (+ 1992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.096163 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.096163 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.096169 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.097828 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.098218 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.098219 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.098220 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.098244 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.098439 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.098440 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.098440 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.098522 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.098605 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.098861 (+ 256us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.098930 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260201 03:08:06.099047 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.091542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.1:49353)
0201 03:08:06.091695 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.091717 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.092134 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.092326 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.092329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.092566 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.092822 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.092824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.093450 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.093451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.093858 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.093859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.094025 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.094476 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.094509 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.096076 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.097913 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.097916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.097918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.098160 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.098304 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.098304 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.098305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.098349 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.098629 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.098632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.098750 (+ 118us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.098833 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.098918 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260201 03:08:06.099463 30734 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43563
I20260201 03:08:06.099535 30734 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:06.099696 30734 heartbeater.cc:507] Master 127.28.181.62:43563 requested a full tablet report, sending...
I20260201 03:08:06.100081 30528 ts_manager.cc:194] Registered new tserver with Master: c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1:36007)
I20260201 03:08:06.100256 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30603
I20260201 03:08:06.100318 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:06.100459 29396 external_mini_cluster.cc:1468] Setting key 20b0fbb301327b57519f0c1bc85bd94d
I20260201 03:08:06.100891 30528 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:49353
I20260201 03:08:06.105337 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.100365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:41503 (local address 127.28.181.62:43563)
0201 03:08:06.101036 (+ 671us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.101037 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.101045 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:06.101061 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.101061 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.101070 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.101119 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.101584 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.102012 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.102570 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.102743 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.103258 (+ 515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.103261 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.103261 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.103268 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.104463 (+ 1195us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.104790 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.104791 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.104791 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.104806 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.104958 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.104958 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.104959 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.105017 (+ 58us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.105056 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.105222 (+ 166us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.105274 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655}
I20260201 03:08:06.105377 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.100324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.1:41503)
0201 03:08:06.100432 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.100440 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.100488 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.101164 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.101165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.101308 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.101513 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.101515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.102080 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.102082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.102497 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.102498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.102551 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.102968 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.102971 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.103175 (+ 204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.104540 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.104541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.104542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.104720 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.104869 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.104870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.104871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.104904 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.105072 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.105073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.105116 (+ 43us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.105186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.105278 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:06.137400 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:06.204370 30745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:06.204501 30745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:06.204520 30745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:06.205832 30745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:06.205868 30745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:06.205888 30745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:06.205941 30745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:06.207329 30745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:06.207463 30745 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:06.207640 30745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:06.209719 30750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.209735 30751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.209719 30753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:06.209889 30745 server_base.cc:1060] running on GCE node
I20260201 03:08:06.210062 30745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:06.210245 30745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:06.211378 30745 hybrid_clock.cc:648] HybridClock initialized: now 1769915286211368 us; error 22 us; skew 500 ppm
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:06.213534 30745 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:06.214035 30745 webserver.cc:492] Webserver started at http://127.28.181.2:44443/ using document root <none> and password file <none>
I20260201 03:08:06.214200 30745 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:06.214247 30745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:06.214362 30745 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:06.215154 30745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "3e27f1a6d825459fb8e84ed6e5903863"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "5153a149e74abb85a8d2872be1ada0f9"
server_key_iv: "ea375f276e5398a4819adab06ffe1f41"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.215433 30745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "3e27f1a6d825459fb8e84ed6e5903863"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "5153a149e74abb85a8d2872be1ada0f9"
server_key_iv: "ea375f276e5398a4819adab06ffe1f41"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.216507 30745 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:06.217060 30760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.217201 30745 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:06.217272 30745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "3e27f1a6d825459fb8e84ed6e5903863"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "5153a149e74abb85a8d2872be1ada0f9"
server_key_iv: "ea375f276e5398a4819adab06ffe1f41"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.217336 30745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:06.237191 30745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:06.238654 30745 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:06.238778 30745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:06.239251 30745 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:06.239297 30745 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.239357 30745 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:06.239398 30745 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:06.246160 30745 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:42849
I20260201 03:08:06.246469 30745 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:06.246891 30875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:42849 every 8 connection(s)
I20260201 03:08:06.247886 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.239565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:54567 (local address 127.28.181.62:43563)
0201 03:08:06.239620 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.239622 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.239764 (+ 142us) server_negotiation.cc:408] Connection header received
0201 03:08:06.240329 (+ 565us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.240330 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.240339 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.240394 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.241389 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.242162 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.242958 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.243166 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.245194 (+ 2028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.245198 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.245198 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.245205 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.246702 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.247142 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.247143 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.247144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.247165 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.247360 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.247361 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.247361 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.247446 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.247509 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.247727 (+ 218us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.247807 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260201 03:08:06.247957 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.239529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.2:54567)
0201 03:08:06.239654 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.239684 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.240248 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.240443 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.240446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.240841 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.241263 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.241268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.242270 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.242271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.242870 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.242872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.243031 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.243634 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.243657 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.245125 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.246827 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.246830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.246832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.247056 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.247227 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.247228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.247228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.247303 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.247501 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.247504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.247618 (+ 114us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.247711 (+ 93us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.247816 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260201 03:08:06.248504 30876 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43563
I20260201 03:08:06.248576 30876 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:06.248759 30876 heartbeater.cc:507] Master 127.28.181.62:43563 requested a full tablet report, sending...
I20260201 03:08:06.249117 30528 ts_manager.cc:194] Registered new tserver with Master: 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849)
I20260201 03:08:06.249591 30528 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:54567
I20260201 03:08:06.250749 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30745
I20260201 03:08:06.250825 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:06.250962 29396 external_mini_cluster.cc:1468] Setting key 7b798b63cd6091af82f8ad01cb878ad3
I20260201 03:08:06.253983 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.249411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.2:39609)
0201 03:08:06.249549 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.249557 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.249620 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.249749 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.249750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.249944 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.250128 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.250130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.250642 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.250643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.251097 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.251104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.251175 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.251555 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.251559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.251768 (+ 209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.253237 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.253238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.253238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.253361 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.253503 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.253503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.253504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.253525 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.253732 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.253733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.253761 (+ 28us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.253852 (+ 91us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.253918 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260201 03:08:06.254078 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.249462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:39609 (local address 127.28.181.62:43563)
0201 03:08:06.249496 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.249498 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.249627 (+ 129us) server_negotiation.cc:408] Connection header received
0201 03:08:06.249660 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.249661 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.249669 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.249712 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.250197 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.250557 (+ 360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.251166 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.251311 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.251846 (+ 535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.251849 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.251849 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.251854 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.253166 (+ 1312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.253425 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.253426 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.253427 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.253449 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.253564 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.253565 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.253565 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.253686 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.253738 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.253881 (+ 143us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.253956 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:06.285315 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:06.350728 30887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:06.350885 30887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:06.350914 30887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:06.352061 30887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:06.352121 30887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:06.352151 30887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:06.352193 30887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:06.353498 30887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:06.353724 30887 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:06.353916 30887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:06.356033 30895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.356091 30892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:06.356202 30893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:06.356846 30887 server_base.cc:1060] running on GCE node
I20260201 03:08:06.357002 30887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:06.357203 30887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:06.358361 30887 hybrid_clock.cc:648] HybridClock initialized: now 1769915286358312 us; error 29 us; skew 500 ppm
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:06.360594 30887 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:06.361008 30887 webserver.cc:492] Webserver started at http://127.28.181.3:46659/ using document root <none> and password file <none>
I20260201 03:08:06.361131 30887 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:06.361156 30887 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:06.361219 30887 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:06.362016 30887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "204b31bac29041e0850177e60f670e78"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "4aed6b5d6e6d12a8f857c840fab3949b"
server_key_iv: "52c494db8f8861a03a43125fd4184b8f"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.362244 30887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "204b31bac29041e0850177e60f670e78"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "4aed6b5d6e6d12a8f857c840fab3949b"
server_key_iv: "52c494db8f8861a03a43125fd4184b8f"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.363361 30887 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:06.363978 30902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.364148 30887 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:06.364209 30887 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "204b31bac29041e0850177e60f670e78"
format_stamp: "Formatted at 2026-02-01 03:08:06 on dist-test-slave-ck1n"
server_key: "4aed6b5d6e6d12a8f857c840fab3949b"
server_key_iv: "52c494db8f8861a03a43125fd4184b8f"
server_key_version: "encryptionkey@0"
I20260201 03:08:06.364264 30887 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:06.377251 30887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:06.378573 30887 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:06.378657 30887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:06.379029 30887 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:06.379058 30887 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.379076 30887 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:06.379087 30887 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.384763 30887 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:44479
I20260201 03:08:06.384797 31017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:44479 every 8 connection(s)
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:06.385046 30887 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:06.387300 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.379389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:50067 (local address 127.28.181.62:43563)
0201 03:08:06.379453 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.379455 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.379564 (+ 109us) server_negotiation.cc:408] Connection header received
0201 03:08:06.379987 (+ 423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.379989 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.380000 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.380042 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.380931 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.381607 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.382371 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.382540 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.385313 (+ 2773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.385317 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.385317 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.385324 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.386469 (+ 1145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.386748 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.386749 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.386749 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.386772 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.386912 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.386912 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.386912 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.386980 (+ 68us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.387020 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.387192 (+ 172us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.387245 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:08:06.387355 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.379348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.3:50067)
0201 03:08:06.379476 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.379500 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.379897 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.380091 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.380094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.380422 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.380847 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.380850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.381696 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.381697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.382282 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.382284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.382423 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.382940 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.382962 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.385210 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.386525 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.386529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.386531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.386691 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.386825 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.386825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.386826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.386868 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.387020 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.387023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.387105 (+ 82us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.387172 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.387250 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":82,"threads_started":1}
I20260201 03:08:06.387881 31018 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43563
I20260201 03:08:06.387945 31018 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:06.388114 31018 heartbeater.cc:507] Master 127.28.181.62:43563 requested a full tablet report, sending...
I20260201 03:08:06.388471 30529 ts_manager.cc:194] Registered new tserver with Master: 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.388722 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 30887
I20260201 03:08:06.388784 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:06.388949 29396 external_mini_cluster.cc:1468] Setting key 60c7417744473882d27de26ad099beb1
I20260201 03:08:06.388952 30529 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:50067
I20260201 03:08:06.389626 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:08:06.393275 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.388820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:35691 (local address 127.28.181.62:43563)
0201 03:08:06.388852 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.388853 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.389107 (+ 254us) server_negotiation.cc:408] Connection header received
0201 03:08:06.389167 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.389168 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.389177 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.389222 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.389610 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.390099 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.390733 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.390826 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.391453 (+ 627us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.391456 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.391456 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.391461 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.392437 (+ 976us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.392711 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.392712 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.392712 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.392737 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.392865 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.392865 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.392866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.392933 (+ 67us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.392995 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.393099 (+ 104us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.393191 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:08:06.393348 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.388898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.28.181.3:35691)
0201 03:08:06.389052 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.389062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.389117 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.389223 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.389223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.389354 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.389547 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.389549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.390168 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.390169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.390664 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.390666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.390732 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.391082 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.391087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.391376 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.392523 (+ 1147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.392524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.392524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.392652 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.392793 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.392793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.392794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.392818 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.392988 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.392988 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.393021 (+ 33us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.393081 (+ 60us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.393233 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:06.407837 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.402686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.28.181.62:43563)
0201 03:08:06.402742 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.402743 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.402753 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:06.402831 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.402832 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.402840 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.402895 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.403497 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.403895 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.404394 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.404560 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.406096 (+ 1536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.406098 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.406098 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.406110 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.407010 (+ 900us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.407283 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.407284 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.407284 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.407308 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.407436 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.407437 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.407437 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.407506 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.407594 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.407679 (+ 85us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.407746 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:08:06.409001 30529 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:06.410161 30529 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:06.414278 31033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.411833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:42849 (local address 127.0.0.1:46806)
0201 03:08:06.411997 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.412022 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.412085 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.412411 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.412411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.412413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:06.412533 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.412534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413401 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.413402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.414056 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.414058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.414121 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.414148 (+ 27us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.414186 (+ 38us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.414220 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":54,"threads_started":1}
I20260201 03:08:06.414595 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.411873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:36007 (local address 127.0.0.1:43832)
0201 03:08:06.412085 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.412095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.412147 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.412320 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.412322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.412324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:06.412530 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.412533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413686 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.413687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.414402 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.414404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.414462 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.414466 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.414494 (+ 28us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.414544 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260201 03:08:06.414820 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.412013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.28.181.2:42849)
0201 03:08:06.412253 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.412267 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.412291 (+ 24us) server_negotiation.cc:408] Connection header received
0201 03:08:06.412324 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.412326 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.412340 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.412405 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:06.412605 (+ 200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413331 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.414148 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.414636 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.414677 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.414702 (+ 25us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.414740 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":59,"threads_started":1}
I20260201 03:08:06.414913 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.411889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:44479 (local address 127.0.0.1:35168)
0201 03:08:06.412123 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.412130 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.412169 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.412743 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.412744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.412746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:06.412915 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.412916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413955 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.413956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.414699 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.414701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.414775 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.414779 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.414806 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.414842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":62,"threads_started":1}
I20260201 03:08:06.415184 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.411947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.28.181.1:36007)
0201 03:08:06.412191 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.412197 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.412204 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:06.412231 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.412233 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.412243 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.412290 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:06.412611 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413614 (+ 1003us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.414491 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.415028 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.415062 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.415086 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.415117 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
I20260201 03:08:06.415324 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.412033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.28.181.3:44479)
0201 03:08:06.412529 (+ 496us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.412535 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.412543 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:06.412650 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.412655 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.412670 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.412730 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:06.412984 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.413890 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.414802 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.415180 (+ 378us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.415210 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.415233 (+ 23us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.415268 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":75,"threads_started":1}
I20260201 03:08:06.415541 30810 tablet_service.cc:1505] Processing CreateTablet for tablet d39760405b0f4eab861e7f7555cea6bd (DEFAULT_TABLE table=test-table [id=5cb2df0e60824b7b84df84ced4b0b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.415601 30657 tablet_service.cc:1505] Processing CreateTablet for tablet d39760405b0f4eab861e7f7555cea6bd (DEFAULT_TABLE table=test-table [id=5cb2df0e60824b7b84df84ced4b0b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.415719 30952 tablet_service.cc:1505] Processing CreateTablet for tablet d39760405b0f4eab861e7f7555cea6bd (DEFAULT_TABLE table=test-table [id=5cb2df0e60824b7b84df84ced4b0b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.415808 30810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39760405b0f4eab861e7f7555cea6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.415838 30657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39760405b0f4eab861e7f7555cea6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.415902 30952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39760405b0f4eab861e7f7555cea6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.417591 31038 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Bootstrap starting.
I20260201 03:08:06.418183 31038 tablet_bootstrap.cc:654] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.418169 31039 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Bootstrap starting.
I20260201 03:08:06.418391 31040 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Bootstrap starting.
I20260201 03:08:06.418689 31038 log.cc:826] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:06.418706 31039 tablet_bootstrap.cc:654] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.418912 31040 tablet_bootstrap.cc:654] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.419205 31039 log.cc:826] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:06.419332 31038 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: No bootstrap required, opened a new log
I20260201 03:08:06.419404 31038 ts_tablet_manager.cc:1403] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260201 03:08:06.419488 31040 log.cc:826] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:06.419872 31039 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: No bootstrap required, opened a new log
I20260201 03:08:06.419924 31039 ts_tablet_manager.cc:1403] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:06.420032 31040 tablet_bootstrap.cc:492] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: No bootstrap required, opened a new log
I20260201 03:08:06.420066 31040 ts_tablet_manager.cc:1403] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:06.420794 31038 raft_consensus.cc:359] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.420905 31038 raft_consensus.cc:385] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.420928 31038 raft_consensus.cc:740] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.421005 31038 consensus_queue.cc:260] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.421187 31038 ts_tablet_manager.cc:1434] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:06.421238 30734 heartbeater.cc:499] Master 127.28.181.62:43563 was elected leader, sending a full tablet report...
I20260201 03:08:06.421200 31040 raft_consensus.cc:359] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.421229 31039 raft_consensus.cc:359] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.421309 31040 raft_consensus.cc:385] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.421336 31040 raft_consensus.cc:740] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e27f1a6d825459fb8e84ed6e5903863, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.421339 31039 raft_consensus.cc:385] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.421355 31039 raft_consensus.cc:740] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204b31bac29041e0850177e60f670e78, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.421408 31040 consensus_queue.cc:260] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.421433 31039 consensus_queue.cc:260] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.421618 31039 ts_tablet_manager.cc:1434] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:06.421620 30876 heartbeater.cc:499] Master 127.28.181.62:43563 was elected leader, sending a full tablet report...
I20260201 03:08:06.421605 31040 ts_tablet_manager.cc:1434] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:06.421700 31018 heartbeater.cc:499] Master 127.28.181.62:43563 was elected leader, sending a full tablet report...
W20260201 03:08:06.497001 30877 tablet.cc:2378] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:06.540637 31044 raft_consensus.cc:493] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:06.540737 31044 raft_consensus.cc:515] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.541062 31044 leader_election.cc:290] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.543038 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.541239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:44479 (local address 127.28.181.1:36965)
0201 03:08:06.541292 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.541302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.541341 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.541494 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.541495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.541498 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:06.541687 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.541689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542320 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.542321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.542814 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.542816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542901 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.542904 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.542933 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.542967 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260201 03:08:06.543057 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.541239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:42849 (local address 127.28.181.1:52203)
0201 03:08:06.541408 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.541418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.541472 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.541574 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.541575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.541576 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:06.541758 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.541760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542362 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.542363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.542836 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.542837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542912 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.542915 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.542945 (+ 30us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.542977 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":12,"thread_start_us":97,"threads_started":1}
I20260201 03:08:06.543426 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.541289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:36965 (local address 127.28.181.3:44479)
0201 03:08:06.541328 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.541331 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.541340 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:06.541404 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.541405 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.541414 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.541478 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:06.541756 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542244 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.542898 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.543320 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.543327 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.543348 (+ 21us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.543378 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260201 03:08:06.543452 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.541306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:52203 (local address 127.28.181.2:42849)
0201 03:08:06.541349 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.541351 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.541487 (+ 136us) server_negotiation.cc:408] Connection header received
0201 03:08:06.541513 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.541514 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.541522 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.541575 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:06.541875 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.542283 (+ 408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.542920 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.543366 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.543372 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.543389 (+ 17us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.543410 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260201 03:08:06.543710 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d39760405b0f4eab861e7f7555cea6bd" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863" is_pre_election: true
I20260201 03:08:06.543735 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d39760405b0f4eab861e7f7555cea6bd" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78" is_pre_election: true
I20260201 03:08:06.543824 30825 raft_consensus.cc:2468] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.543884 30972 raft_consensus.cc:2468] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.544015 30622 leader_election.cc:304] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e27f1a6d825459fb8e84ed6e5903863, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.544142 31044 raft_consensus.cc:2804] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:06.544188 31044 raft_consensus.cc:493] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:06.544209 31044 raft_consensus.cc:3060] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.544798 31044 raft_consensus.cc:515] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.544921 31044 leader_election.cc:290] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.545068 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d39760405b0f4eab861e7f7555cea6bd" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78"
I20260201 03:08:06.545073 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d39760405b0f4eab861e7f7555cea6bd" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863"
I20260201 03:08:06.545131 30972 raft_consensus.cc:3060] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.545131 30825 raft_consensus.cc:3060] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.545928 30825 raft_consensus.cc:2468] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
I20260201 03:08:06.545928 30972 raft_consensus.cc:2468] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
I20260201 03:08:06.546109 30621 leader_election.cc:304] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204b31bac29041e0850177e60f670e78, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.546229 31044 raft_consensus.cc:2804] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:06.546346 31044 raft_consensus.cc:697] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 LEADER]: Becoming Leader. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Running, Role: LEADER
I20260201 03:08:06.546449 31044 consensus_queue.cc:237] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.547196 30528 catalog_manager.cc:5654] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:06.562673 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.560729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.28.181.1:36007)
0201 03:08:06.560793 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.560794 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.560803 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:06.560923 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.560924 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.560936 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.560995 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:06.561214 (+ 219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.561637 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.562215 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.562360 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.562385 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:06.562453 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:06.562530 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.562581 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.562601 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:08:06.564059 30825 raft_consensus.cc:1275] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.564059 30972 raft_consensus.cc:1275] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.564358 31048 consensus_queue.cc:1048] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.564483 31044 consensus_queue.cc:1048] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.567046 31053 mvcc.cc:204] Tried to move back new op lower bound from 7249573013764124672 to 7249573013694849024. Current Snapshot: MvccSnapshot[applied={T|T < 7249573013764124672}]
I20260201 03:08:06.568614 30529 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:06.568678 30529 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:06.569290 30529 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Sending DeleteTablet for 3 replicas of tablet d39760405b0f4eab861e7f7555cea6bd
I20260201 03:08:06.569532 30810 tablet_service.cc:1552] Processing DeleteTablet for tablet d39760405b0f4eab861e7f7555cea6bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:46806
I20260201 03:08:06.569545 30952 tablet_service.cc:1552] Processing DeleteTablet for tablet d39760405b0f4eab861e7f7555cea6bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:35168
I20260201 03:08:06.569722 31060 tablet_replica.cc:333] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: stopping tablet replica
I20260201 03:08:06.569804 30657 tablet_service.cc:1552] Processing DeleteTablet for tablet d39760405b0f4eab861e7f7555cea6bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:43832
I20260201 03:08:06.569832 31060 raft_consensus.cc:2243] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:06.569932 31060 raft_consensus.cc:2272] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.569988 30529 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260201 03:08:06.570039 31062 tablet_replica.cc:333] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: stopping tablet replica
W20260201 03:08:06.570115 30529 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:06.570258 31062 raft_consensus.cc:2243] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:06.570303 31060 ts_tablet_manager.cc:1916] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.570434 31062 raft_consensus.cc:2272] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.570787 31062 ts_tablet_manager.cc:1916] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.571251 30810 tablet_service.cc:1505] Processing CreateTablet for tablet 080b508e229e4e218338cf543fc9a214 (DEFAULT_TABLE table=test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.571344 30810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080b508e229e4e218338cf543fc9a214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.571864 31060 ts_tablet_manager.cc:1929] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:06.571929 31060 log.cc:1199] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/d39760405b0f4eab861e7f7555cea6bd
I20260201 03:08:06.572136 31060 ts_tablet_manager.cc:1950] T d39760405b0f4eab861e7f7555cea6bd P 3e27f1a6d825459fb8e84ed6e5903863: Deleting consensus metadata
I20260201 03:08:06.572444 31062 ts_tablet_manager.cc:1929] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:06.572444 30516 catalog_manager.cc:4985] TS 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849): tablet d39760405b0f4eab861e7f7555cea6bd (table test-table [id=5cb2df0e60824b7b84df84ced4b0b832]) successfully deleted
I20260201 03:08:06.572503 31040 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Bootstrap starting.
I20260201 03:08:06.572535 30952 tablet_service.cc:1505] Processing CreateTablet for tablet 080b508e229e4e218338cf543fc9a214 (DEFAULT_TABLE table=test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.572568 31062 log.cc:1199] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/d39760405b0f4eab861e7f7555cea6bd
I20260201 03:08:06.572630 30952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080b508e229e4e218338cf543fc9a214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.572798 31062 ts_tablet_manager.cc:1950] T d39760405b0f4eab861e7f7555cea6bd P c7cc69635c5c4e6d93887f01ef211ad3: Deleting consensus metadata
I20260201 03:08:06.572890 31040 tablet_bootstrap.cc:654] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.573623 31040 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: No bootstrap required, opened a new log
I20260201 03:08:06.573671 31040 ts_tablet_manager.cc:1403] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:06.573817 31039 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Bootstrap starting.
I20260201 03:08:06.573817 31040 raft_consensus.cc:359] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.573884 31040 raft_consensus.cc:385] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.573904 31040 raft_consensus.cc:740] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e27f1a6d825459fb8e84ed6e5903863, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.573942 31040 consensus_queue.cc:260] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.573958 30514 catalog_manager.cc:4985] TS c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1:36007): tablet d39760405b0f4eab861e7f7555cea6bd (table test-table [id=5cb2df0e60824b7b84df84ced4b0b832]) successfully deleted
I20260201 03:08:06.574008 31040 ts_tablet_manager.cc:1434] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:06.574204 31039 tablet_bootstrap.cc:654] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.574295 30657 tablet_service.cc:1505] Processing CreateTablet for tablet 080b508e229e4e218338cf543fc9a214 (DEFAULT_TABLE table=test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:06.574393 30657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080b508e229e4e218338cf543fc9a214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.574879 31039 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: No bootstrap required, opened a new log
I20260201 03:08:06.574923 31039 ts_tablet_manager.cc:1403] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:06.575044 31039 raft_consensus.cc:359] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.575095 31039 raft_consensus.cc:385] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.575114 31039 raft_consensus.cc:740] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204b31bac29041e0850177e60f670e78, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.575155 31039 consensus_queue.cc:260] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.575223 31039 ts_tablet_manager.cc:1434] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.575521 31038 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Bootstrap starting.
I20260201 03:08:06.575901 31038 tablet_bootstrap.cc:654] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.575986 31061 tablet_replica.cc:333] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: stopping tablet replica
I20260201 03:08:06.576056 31061 raft_consensus.cc:2243] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:06.576129 31061 raft_consensus.cc:2272] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.576434 31061 ts_tablet_manager.cc:1916] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.576603 31038 tablet_bootstrap.cc:492] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: No bootstrap required, opened a new log
I20260201 03:08:06.576650 31038 ts_tablet_manager.cc:1403] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:06.576771 31038 raft_consensus.cc:359] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.576824 31038 raft_consensus.cc:385] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.576841 31038 raft_consensus.cc:740] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.576890 31038 consensus_queue.cc:260] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.576980 31038 ts_tablet_manager.cc:1434] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:06.577839 31061 ts_tablet_manager.cc:1929] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:06.577898 31061 log.cc:1199] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/d39760405b0f4eab861e7f7555cea6bd
I20260201 03:08:06.578094 31061 ts_tablet_manager.cc:1950] T d39760405b0f4eab861e7f7555cea6bd P 204b31bac29041e0850177e60f670e78: Deleting consensus metadata
I20260201 03:08:06.578433 30515 catalog_manager.cc:4985] TS 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479): tablet d39760405b0f4eab861e7f7555cea6bd (table test-table [id=5cb2df0e60824b7b84df84ced4b0b832]) successfully deleted
I20260201 03:08:06.596875 31048 raft_consensus.cc:493] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:06.596935 31048 raft_consensus.cc:515] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.597059 31048 leader_election.cc:290] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.597170 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080b508e229e4e218338cf543fc9a214" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78" is_pre_election: true
I20260201 03:08:06.597262 30972 raft_consensus.cc:2468] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.597239 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080b508e229e4e218338cf543fc9a214" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863" is_pre_election: true
I20260201 03:08:06.597291 30825 raft_consensus.cc:2468] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.597373 30621 leader_election.cc:304] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204b31bac29041e0850177e60f670e78, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.597446 31048 raft_consensus.cc:2804] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:06.597491 31048 raft_consensus.cc:493] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:06.597503 31048 raft_consensus.cc:3060] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.598119 31048 raft_consensus.cc:515] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.598233 31048 leader_election.cc:290] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.598354 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080b508e229e4e218338cf543fc9a214" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863"
I20260201 03:08:06.598416 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080b508e229e4e218338cf543fc9a214" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78"
I20260201 03:08:06.598444 30825 raft_consensus.cc:3060] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.598477 30972 raft_consensus.cc:3060] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.598949 30825 raft_consensus.cc:2468] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
W20260201 03:08:06.598948 30735 tablet.cc:2378] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:06.599088 30972 raft_consensus.cc:2468] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
I20260201 03:08:06.599094 30622 leader_election.cc:304] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e27f1a6d825459fb8e84ed6e5903863, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.599186 31048 raft_consensus.cc:2804] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:06.599234 31048 raft_consensus.cc:697] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 LEADER]: Becoming Leader. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Running, Role: LEADER
I20260201 03:08:06.599267 31048 consensus_queue.cc:237] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.599736 30529 catalog_manager.cc:5654] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } health_report { overall_health: UNKNOWN } } }
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915285, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:06.628347 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.622535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.28.181.62:43563)
0201 03:08:06.622590 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.622592 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.622671 (+ 79us) server_negotiation.cc:408] Connection header received
0201 03:08:06.622726 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.622727 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.622737 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.622786 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.623296 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.624026 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.624498 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.624643 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.626264 (+ 1621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.626267 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.626267 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.626273 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.627482 (+ 1209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.627847 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.627847 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.627848 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.627864 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.627994 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.627995 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.627995 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.628059 (+ 64us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.628100 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.628235 (+ 135us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.628292 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:06.628449 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.622467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43563 (local address 127.0.0.1:41276)
0201 03:08:06.622603 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.622613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.622673 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.622800 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.622801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.623067 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:06.623224 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.623226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.624106 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.624106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.624433 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.624434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.624480 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.624932 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:06.624935 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:06.626185 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:06.627555 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.627561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.627563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.627778 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.627916 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:06.627916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:06.627917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:06.627946 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:06.628106 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:06.628111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:06.628154 (+ 43us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.628217 (+ 63us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.628349 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260201 03:08:06.630118 30529 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:41276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:08:06.630309 30529 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:06.631484 30657 tablet_service.cc:1505] Processing CreateTablet for tablet aff69d19127b44679f60c5100ff0a166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8726390e3924d4c8ce917f1e64a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:06.631570 30952 tablet_service.cc:1505] Processing CreateTablet for tablet aff69d19127b44679f60c5100ff0a166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8726390e3924d4c8ce917f1e64a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:06.631623 30657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff69d19127b44679f60c5100ff0a166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.631665 30952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff69d19127b44679f60c5100ff0a166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.632783 31038 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: Bootstrap starting.
I20260201 03:08:06.632783 31039 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: Bootstrap starting.
I20260201 03:08:06.632828 30810 tablet_service.cc:1505] Processing CreateTablet for tablet aff69d19127b44679f60c5100ff0a166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8726390e3924d4c8ce917f1e64a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:06.632929 30810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff69d19127b44679f60c5100ff0a166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:06.633152 31038 tablet_bootstrap.cc:654] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.633198 31039 tablet_bootstrap.cc:654] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.634233 31038 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: No bootstrap required, opened a new log
I20260201 03:08:06.634290 31038 ts_tablet_manager.cc:1403] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260201 03:08:06.634418 31040 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: Bootstrap starting.
I20260201 03:08:06.634426 31038 raft_consensus.cc:359] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.634480 31038 raft_consensus.cc:385] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.634514 31038 raft_consensus.cc:740] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.634563 31038 consensus_queue.cc:260] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.634652 31038 ts_tablet_manager.cc:1434] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:06.634723 31048 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.634764 31048 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.634861 31040 tablet_bootstrap.cc:654] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:06.635073 31039 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: No bootstrap required, opened a new log
I20260201 03:08:06.635113 31039 ts_tablet_manager.cc:1403] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:06.635244 31039 raft_consensus.cc:359] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.635293 31039 raft_consensus.cc:385] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.635308 31039 raft_consensus.cc:740] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204b31bac29041e0850177e60f670e78, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.635346 31039 consensus_queue.cc:260] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.635422 31039 ts_tablet_manager.cc:1434] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:06.635461 31040 tablet_bootstrap.cc:492] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: No bootstrap required, opened a new log
I20260201 03:08:06.635505 31040 ts_tablet_manager.cc:1403] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:06.635615 31046 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.635660 31040 raft_consensus.cc:359] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.635705 31046 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.635715 31040 raft_consensus.cc:385] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:06.635733 31040 raft_consensus.cc:740] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e27f1a6d825459fb8e84ed6e5903863, State: Initialized, Role: FOLLOWER
I20260201 03:08:06.635768 31040 consensus_queue.cc:260] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.635831 31040 ts_tablet_manager.cc:1434] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260201 03:08:06.635933 31019 tablet.cc:2378] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:06.636168 31045 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.636231 31045 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.809082 31048 raft_consensus.cc:493] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:06.809175 31048 raft_consensus.cc:515] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.809309 31048 leader_election.cc:290] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.809492 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aff69d19127b44679f60c5100ff0a166" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78" is_pre_election: true
I20260201 03:08:06.809511 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aff69d19127b44679f60c5100ff0a166" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863" is_pre_election: true
I20260201 03:08:06.809584 30972 raft_consensus.cc:2468] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.809584 30825 raft_consensus.cc:2468] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 0.
I20260201 03:08:06.809707 30621 leader_election.cc:304] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204b31bac29041e0850177e60f670e78, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.809808 31048 raft_consensus.cc:2804] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:06.809842 31048 raft_consensus.cc:493] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:06.809865 31048 raft_consensus.cc:3060] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.810362 31048 raft_consensus.cc:515] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.810485 31048 leader_election.cc:290] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849), 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479)
I20260201 03:08:06.810602 30825 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aff69d19127b44679f60c5100ff0a166" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e27f1a6d825459fb8e84ed6e5903863"
I20260201 03:08:06.810631 30972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aff69d19127b44679f60c5100ff0a166" candidate_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204b31bac29041e0850177e60f670e78"
I20260201 03:08:06.810672 30825 raft_consensus.cc:3060] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.810678 30972 raft_consensus.cc:3060] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:06.811316 30972 raft_consensus.cc:2468] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
I20260201 03:08:06.811316 30825 raft_consensus.cc:2468] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cc69635c5c4e6d93887f01ef211ad3 in term 1.
I20260201 03:08:06.811502 30622 leader_election.cc:304] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e27f1a6d825459fb8e84ed6e5903863, c7cc69635c5c4e6d93887f01ef211ad3; no voters:
I20260201 03:08:06.811604 31048 raft_consensus.cc:2804] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:06.811684 31048 raft_consensus.cc:697] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 LEADER]: Becoming Leader. State: Replica: c7cc69635c5c4e6d93887f01ef211ad3, State: Running, Role: LEADER
I20260201 03:08:06.811734 31048 consensus_queue.cc:237] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } }
I20260201 03:08:06.811950 31044 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: TxnStatusTablet state changed. Reason: New leader c7cc69635c5c4e6d93887f01ef211ad3. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.812026 31044 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:06.812167 31077 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:08:06.812249 31077 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260201 03:08:06.812268 30529 catalog_manager.cc:5654] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:06.817050 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.815261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.28.181.1:36007)
0201 03:08:06.815309 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.815310 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.815322 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:08:06.815353 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.815354 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.815364 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.815411 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:06.815629 (+ 218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.816113 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.816634 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.816787 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.816803 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:06.816823 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:06.816882 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.816976 (+ 94us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.816995 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:08:06.817085 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.815086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:36007 (local address 127.0.0.1:52954)
0201 03:08:06.815155 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.815170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.815209 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.815422 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.815422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.815424 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:06.815564 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.815565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.816182 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.816183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.816545 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.816546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.816615 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.816633 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:06.816909 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:06.816919 (+ 10us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.816972 (+ 53us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.817021 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260201 03:08:06.817938 30972 raft_consensus.cc:1275] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.817950 30825 raft_consensus.cc:1275] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.818147 31044 consensus_queue.cc:1048] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.818241 31048 consensus_queue.cc:1048] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.818684 31046 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: TxnStatusTablet state changed. Reason: New leader c7cc69635c5c4e6d93887f01ef211ad3. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.818758 31046 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.818763 31045 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: TxnStatusTablet state changed. Reason: New leader c7cc69635c5c4e6d93887f01ef211ad3. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.818831 31045 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.819555 31048 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.819633 31048 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:06.819698 31051 mvcc.cc:204] Tried to move back new op lower bound from 7249573014804881408 to 7249573014781116416. Current Snapshot: MvccSnapshot[applied={T|T < 7249573014804881408}]
I20260201 03:08:06.819761 31046 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.819832 31046 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 204b31bac29041e0850177e60f670e78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.820003 31052 mvcc.cc:204] Tried to move back new op lower bound from 7249573014804881408 to 7249573014781116416. Current Snapshot: MvccSnapshot[applied={T|T < 7249573014804881408}]
I20260201 03:08:06.820158 31045 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.820236 31045 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P 3e27f1a6d825459fb8e84ed6e5903863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:06.819561 31044 tablet_replica.cc:442] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7cc69635c5c4e6d93887f01ef211ad3" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 36007 } } peers { permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 } } peers { permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 } } }
I20260201 03:08:06.820963 31044 tablet_replica.cc:445] T aff69d19127b44679f60c5100ff0a166 P c7cc69635c5c4e6d93887f01ef211ad3: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:06.824738 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.822824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49275 (local address 127.28.181.1:36007)
0201 03:08:06.822875 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.822877 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.822990 (+ 113us) server_negotiation.cc:408] Connection header received
0201 03:08:06.823040 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.823040 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.823047 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.823085 (+ 38us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:06.823373 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.823737 (+ 364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.824423 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.824526 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.824538 (+ 12us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:06.824555 (+ 17us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:06.824592 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.824668 (+ 76us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.824681 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:06.824802 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.822775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:36007 (local address 127.28.181.1:49275)
0201 03:08:06.822925 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:06.822935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:06.822991 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:06.823102 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:06.823102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:06.823107 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:06.823309 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.823310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.823808 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.823808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:06.824361 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:06.824362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.824427 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.824439 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:06.824615 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:06.824617 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:06.824660 (+ 43us) client_negotiation.cc:240] Negotiation successful
0201 03:08:06.824695 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260201 03:08:06.826627 30825 raft_consensus.cc:1275] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.826639 30972 raft_consensus.cc:1275] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Refusing update from remote peer c7cc69635c5c4e6d93887f01ef211ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:06.826762 31048 consensus_queue.cc:1048] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e27f1a6d825459fb8e84ed6e5903863" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.826828 31044 consensus_queue.cc:1048] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204b31bac29041e0850177e60f670e78" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:06.835763 30529 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:06.835839 30529 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:06.836334 30529 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 59993cdfc8ce4be3989cb7320cdeb2a6: Sending DeleteTablet for 3 replicas of tablet 080b508e229e4e218338cf543fc9a214
I20260201 03:08:06.836534 30810 tablet_service.cc:1552] Processing DeleteTablet for tablet 080b508e229e4e218338cf543fc9a214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:46806
I20260201 03:08:06.836618 31060 tablet_replica.cc:333] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: stopping tablet replica
I20260201 03:08:06.836685 31060 raft_consensus.cc:2243] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:06.836717 31060 pending_rounds.cc:70] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Trying to abort 2 pending ops.
I20260201 03:08:06.836752 31060 pending_rounds.cc:77] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7249573014840799232 op_type: PARTICIPANT_OP participant_request { tablet_id: "080b508e229e4e218338cf543fc9a214" op { type: BEGIN_TXN txn_id: 0 } }
I20260201 03:08:06.836838 31060 pending_rounds.cc:77] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7249573013910892544 op_type: NO_OP noop_request { }
I20260201 03:08:06.836871 31060 raft_consensus.cc:2889] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260201 03:08:06.836903 31060 raft_consensus.cc:2272] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.837059 30952 tablet_service.cc:1552] Processing DeleteTablet for tablet 080b508e229e4e218338cf543fc9a214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:35168
I20260201 03:08:06.837066 30657 tablet_service.cc:1552] Processing DeleteTablet for tablet 080b508e229e4e218338cf543fc9a214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:06 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:43832
I20260201 03:08:06.837162 31061 tablet_replica.cc:333] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: stopping tablet replica
I20260201 03:08:06.837218 31061 raft_consensus.cc:2243] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:06.837278 31061 raft_consensus.cc:2272] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.837345 31062 tablet_replica.cc:333] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: stopping tablet replica
I20260201 03:08:06.837392 31062 raft_consensus.cc:2243] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:06.837451 31062 raft_consensus.cc:2272] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:06.837795 31061 ts_tablet_manager.cc:1916] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.838999 31061 ts_tablet_manager.cc:1929] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:06.839038 31061 log.cc:1199] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/080b508e229e4e218338cf543fc9a214
I20260201 03:08:06.839228 31061 ts_tablet_manager.cc:1950] T 080b508e229e4e218338cf543fc9a214 P 204b31bac29041e0850177e60f670e78: Deleting consensus metadata
I20260201 03:08:06.839493 30515 catalog_manager.cc:4985] TS 204b31bac29041e0850177e60f670e78 (127.28.181.3:44479): tablet 080b508e229e4e218338cf543fc9a214 (table test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]) successfully deleted
I20260201 03:08:06.839584 31062 ts_tablet_manager.cc:1916] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.840947 31062 ts_tablet_manager.cc:1929] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:06.841005 31062 log.cc:1199] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/080b508e229e4e218338cf543fc9a214
I20260201 03:08:06.841181 31062 ts_tablet_manager.cc:1950] T 080b508e229e4e218338cf543fc9a214 P c7cc69635c5c4e6d93887f01ef211ad3: Deleting consensus metadata
I20260201 03:08:06.841488 30514 catalog_manager.cc:4985] TS c7cc69635c5c4e6d93887f01ef211ad3 (127.28.181.1:36007): tablet 080b508e229e4e218338cf543fc9a214 (table test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]) successfully deleted
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[30481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
W20260201 03:08:06.842002 31086 consensus_queue.cc:1175] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3e27f1a6d825459fb8e84ed6e5903863" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260201 03:08:06.842212 31060 ts_tablet_manager.cc:1916] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:06.843194 31060 ts_tablet_manager.cc:1929] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:06.843247 31060 log.cc:1199] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/080b508e229e4e218338cf543fc9a214
I20260201 03:08:06.843425 31060 ts_tablet_manager.cc:1950] T 080b508e229e4e218338cf543fc9a214 P 3e27f1a6d825459fb8e84ed6e5903863: Deleting consensus metadata
I20260201 03:08:06.843694 30516 catalog_manager.cc:4985] TS 3e27f1a6d825459fb8e84ed6e5903863 (127.28.181.2:42849): tablet 080b508e229e4e218338cf543fc9a214 (table test-table [id=b1047236f40e46bbb9ad0e6e2cbd816a]) successfully deleted
I20260201 03:08:06.845745 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.837302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.28.181.1:36007)
0201 03:08:06.837331 (+ 29us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.837333 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.837688 (+ 355us) server_negotiation.cc:408] Connection header received
0201 03:08:06.837729 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.837730 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.837739 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.837827 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.838554 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.839167 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.839820 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.839959 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.843292 (+ 3333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.843299 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.843300 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.843305 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.844971 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.845232 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.845235 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.845235 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.845268 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.845377 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.845377 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.845378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.845458 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.845565 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.845599 (+ 34us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.845671 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260201 03:08:06.845949 30728 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52970
I20260201 03:08:06.851393 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:06.846686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.28.181.62:43563)
0201 03:08:06.846723 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:06.846725 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:06.846765 (+ 40us) server_negotiation.cc:408] Connection header received
0201 03:08:06.846811 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:06.846812 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:06.846819 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:06.846871 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:06.847321 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.847952 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:06.848369 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:06.848512 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:06.849301 (+ 789us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:06.849304 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:06.849304 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:06.849310 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:06.850659 (+ 1349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.850908 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.850909 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.850909 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.850929 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:06.851037 (+ 108us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:06.851038 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:06.851039 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:06.851119 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:06.851192 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:06.851255 (+ 63us) server_negotiation.cc:300] Negotiation successful
0201 03:08:06.851317 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260201 03:08:06.851601 30529 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41284
I20260201 03:08:06.852027 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30603
I20260201 03:08:06.858090 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30745
I20260201 03:08:06.864455 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30887
I20260201 03:08:06.868180 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 30497
2026-02-01T03:08:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2846 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[31095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[31095](info): set up 2 sockets
Feb 01 03:08:06 dist-test-slave-ck1n krb5kdc[31095](info): commencing operation
krb5kdc: starting...
W20260201 03:08:08.607699 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.720s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:08 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:08.716348 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38061
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:38061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:08.783592 31111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:08.783774 31111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:08.783818 31111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:08.785115 31111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:08.785172 31111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:08:08.785193 31111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:08.785219 31111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:08.785265 31111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:08.785290 31111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:08.786791 31111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:38061
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:08.787009 31111 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:08.787189 31111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:08.789119 31119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:08.789173 31116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:08.789119 31117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:08.789301 31111 server_base.cc:1060] running on GCE node
I20260201 03:08:08.789505 31111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:08.789736 31111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:08.790880 31111 hybrid_clock.cc:648] HybridClock initialized: now 1769915288790872 us; error 26 us; skew 500 ppm
Feb 01 03:08:08 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:08.792732 31111 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:08.793262 31111 webserver.cc:492] Webserver started at http://127.28.181.62:33751/ using document root <none> and password file <none>
I20260201 03:08:08.793462 31111 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:08.793525 31111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:08.793648 31111 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:08.794492 31111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "2a1c3b38f40548d29e3e1af4ae3ec231"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "e7d720de574f945725f7b290e26a6456"
server_key_iv: "743eb4f2d5c8dd965b97ef180ba16eaf"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.794839 31111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "2a1c3b38f40548d29e3e1af4ae3ec231"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "e7d720de574f945725f7b290e26a6456"
server_key_iv: "743eb4f2d5c8dd965b97ef180ba16eaf"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.796032 31111 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260201 03:08:08.796649 31126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:08.796811 31111 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:08.796886 31111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "2a1c3b38f40548d29e3e1af4ae3ec231"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "e7d720de574f945725f7b290e26a6456"
server_key_iv: "743eb4f2d5c8dd965b97ef180ba16eaf"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.796953 31111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:08.807108 31111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:08.811137 31111 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:08.811262 31111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:08.815203 31111 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:38061
I20260201 03:08:08.815243 31188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:38061 every 8 connection(s)
I20260201 03:08:08.815507 31111 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:08.816227 31189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:08.817955 31189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Bootstrap starting.
I20260201 03:08:08.818398 31189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:08.818571 31189 log.cc:826] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:08.819121 31189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: No bootstrap required, opened a new log
I20260201 03:08:08.819927 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31111
I20260201 03:08:08.820016 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:08.820156 29396 external_mini_cluster.cc:1468] Setting key cdfd0af47d65be7d0fdd98bac8404e7c
I20260201 03:08:08.820183 31189 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } }
I20260201 03:08:08.820325 31189 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:08.820369 31189 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1c3b38f40548d29e3e1af4ae3ec231, State: Initialized, Role: FOLLOWER
I20260201 03:08:08.820477 31189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } }
I20260201 03:08:08.820544 31189 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:08.820587 31189 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:08.820636 31189 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:08.821089 31189 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } }
I20260201 03:08:08.821201 31189 leader_election.cc:304] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a1c3b38f40548d29e3e1af4ae3ec231; no voters:
I20260201 03:08:08.821343 31189 leader_election.cc:290] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:08.821389 31194 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:08.821522 31194 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1c3b38f40548d29e3e1af4ae3ec231, State: Running, Role: LEADER
I20260201 03:08:08.821552 31189 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:08.821611 31194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } }
I20260201 03:08:08.821898 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1c3b38f40548d29e3e1af4ae3ec231. Latest consensus state: current_term: 1 leader_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } } }
I20260201 03:08:08.821956 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:08.822822 31196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1c3b38f40548d29e3e1af4ae3ec231" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38061 } } }
I20260201 03:08:08.823439 31196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231 [sys.catalog]: This master's current role is: LEADER
W20260201 03:08:08.823163 31209 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:08.823505 31209 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:08.823552 31199 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:08.823719 31199 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:08.825408 31199 catalog_manager.cc:1357] Generated new cluster ID: 7d0a82a3ef054a43be35b3fd65ce4019
I20260201 03:08:08.825465 31199 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:08:08 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:08.829146 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:08.820519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.28.181.62:38061)
0201 03:08:08.820668 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:08.820679 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:08.820721 (+ 42us) server_negotiation.cc:408] Connection header received
0201 03:08:08.820862 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:08.820866 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:08.820898 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:08.820999 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:08.821882 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.823098 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:08.823859 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.824071 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:08.826239 (+ 2168us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:08.826261 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:08.826262 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:08.826278 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:08.827857 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.828174 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.828177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.828177 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.828222 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.828366 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.828366 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.828367 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.828560 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:08.828661 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:08.828822 (+ 161us) server_negotiation.cc:300] Negotiation successful
0201 03:08:08.829004 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260201 03:08:08.836373 31199 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:08.836833 31199 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:08.841634 31199 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Generated new TSK 0
I20260201 03:08:08.841818 31199 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:08.879738 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38061
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:08.944880 31217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:08.945039 31217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:08.945075 31217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:08.946265 31217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:08.946323 31217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:08.946353 31217 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:08:08.946378 31217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:08.946405 31217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:08.947698 31217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:38061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:08.947916 31217 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:08.948105 31217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:08.950202 31222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:08.950228 31223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:08.950304 31217 server_base.cc:1060] running on GCE node
W20260201 03:08:08.950191 31225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:08.950573 31217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:08.950784 31217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:08.951915 31217 hybrid_clock.cc:648] HybridClock initialized: now 1769915288951906 us; error 28 us; skew 500 ppm
Feb 01 03:08:08 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:08.953962 31217 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:08.954417 31217 webserver.cc:492] Webserver started at http://127.28.181.1:33651/ using document root <none> and password file <none>
I20260201 03:08:08.954612 31217 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:08.954663 31217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:08.954767 31217 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:08.955641 31217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "a25056fed19b44f2902a3462ba9d17d9"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "07b87869e36194689df91620b0e4a0a2"
server_key_iv: "90b489f6d7a15030bc834fee5f949549"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.955948 31217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "a25056fed19b44f2902a3462ba9d17d9"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "07b87869e36194689df91620b0e4a0a2"
server_key_iv: "90b489f6d7a15030bc834fee5f949549"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.956962 31217 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:08.957508 31232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:08.957686 31217 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:08.957760 31217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "a25056fed19b44f2902a3462ba9d17d9"
format_stamp: "Formatted at 2026-02-01 03:08:08 on dist-test-slave-ck1n"
server_key: "07b87869e36194689df91620b0e4a0a2"
server_key_iv: "90b489f6d7a15030bc834fee5f949549"
server_key_version: "encryptionkey@0"
I20260201 03:08:08.957854 31217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:08.984881 31217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:08.986526 31217 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:08.986660 31217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:08.987174 31217 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:08.987215 31217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:08.987262 31217 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:08.987277 31217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:08:08 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:08.993757 31217 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:37471
I20260201 03:08:08.993850 31347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:37471 every 8 connection(s)
I20260201 03:08:08.994093 31217 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:08.994750 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:08.987482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:37375 (local address 127.28.181.62:38061)
0201 03:08:08.987537 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:08.987539 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:08.987673 (+ 134us) server_negotiation.cc:408] Connection header received
0201 03:08:08.988094 (+ 421us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:08.988095 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:08.988110 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:08.988162 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:08.989098 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.989830 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:08.990327 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.990487 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:08.992158 (+ 1671us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:08.992162 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:08.992163 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:08.992171 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:08.993443 (+ 1272us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.993921 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.993922 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.993922 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.993946 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.994161 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.994162 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.994162 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.994277 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:08.994338 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:08.994631 (+ 293us) server_negotiation.cc:300] Negotiation successful
0201 03:08:08.994696 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:08:08.994820 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:08.987446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.1:37375)
0201 03:08:08.987568 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:08.987597 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:08.988007 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:08.988321 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:08.988324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:08.988591 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:08.988974 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:08.988978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.989920 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:08.989920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:08.990245 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:08.990247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.990384 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:08.990811 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:08.990831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:08.992073 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:08.993541 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:08.993546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:08.993548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:08.993846 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:08.994016 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:08.994016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:08.994017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:08.994083 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:08.994365 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:08.994369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:08.994492 (+ 123us) client_negotiation.cc:769] Sending connection context
0201 03:08:08.994589 (+ 97us) client_negotiation.cc:240] Negotiation successful
0201 03:08:08.994675 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260201 03:08:08.995237 31348 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38061
I20260201 03:08:08.995311 31348 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:08.995479 31348 heartbeater.cc:507] Master 127.28.181.62:38061 requested a full tablet report, sending...
I20260201 03:08:08.995872 31142 ts_manager.cc:194] Registered new tserver with Master: a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471)
I20260201 03:08:08.996582 31142 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:37375
I20260201 03:08:08.999722 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:08.996167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49485 (local address 127.28.181.62:38061)
0201 03:08:08.996197 (+ 30us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:08.996199 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:08.996265 (+ 66us) server_negotiation.cc:408] Connection header received
0201 03:08:08.996304 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:08.996305 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:08.996312 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:08.996367 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:08.996647 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.997070 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:08.997484 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.997600 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:08.998074 (+ 474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:08.998077 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:08.998077 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:08.998081 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:08.999127 (+ 1046us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.999332 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.999332 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.999332 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.999346 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:08.999436 (+ 90us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:08.999436 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:08.999436 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:08.999490 (+ 54us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:08.999526 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:08.999624 (+ 98us) server_negotiation.cc:300] Negotiation successful
0201 03:08:08.999676 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:08.999732 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:08.996116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.1:49485)
0201 03:08:08.996220 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:08.996228 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:08.996270 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:08.996369 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:08.996369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:08.996463 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:08.996590 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:08.996590 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.997135 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:08.997136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:08.997428 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:08.997429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:08.997480 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:08.997825 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:08.997828 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:08.998017 (+ 189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:08.999184 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:08.999184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:08.999185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:08.999284 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:08.999389 (+ 105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:08.999390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:08.999390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:08.999407 (+ 17us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:08.999539 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:08.999540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:08.999567 (+ 27us) client_negotiation.cc:769] Sending connection context
0201 03:08:08.999612 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:08:08.999674 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20260201 03:08:09.003185 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31217
I20260201 03:08:09.003262 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:09.003367 29396 external_mini_cluster.cc:1468] Setting key 2d925243c94bbe42b7d33c0a9ace8a88
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:09.038332 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38061
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:09.105109 31359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:09.105237 31359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:09.105257 31359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:09.106444 31359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:09.106480 31359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:09.106498 31359 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:08:09.106508 31359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:09.106521 31359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:09.107738 31359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:38061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:09.107878 31359 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:09.107983 31359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:08:09.109930 31359 server_base.cc:1060] running on GCE node
W20260201 03:08:09.109897 31365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:09.109899 31367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:09.109892 31364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:09.110219 31359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:09.110401 31359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:09.111523 31359 hybrid_clock.cc:648] HybridClock initialized: now 1769915289111507 us; error 32 us; skew 500 ppm
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:09.113339 31359 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:09.113812 31359 webserver.cc:492] Webserver started at http://127.28.181.2:33365/ using document root <none> and password file <none>
I20260201 03:08:09.114020 31359 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:09.114066 31359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:09.114169 31359 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:09.114960 31359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "67d15bb1aceb45f7b36518caf476a68e"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "3174ec9a6c93e7210c35f8fba8ce42b0"
server_key_iv: "7bde335d7839f42640e2afaf1f337f74"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.115231 31359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "67d15bb1aceb45f7b36518caf476a68e"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "3174ec9a6c93e7210c35f8fba8ce42b0"
server_key_iv: "7bde335d7839f42640e2afaf1f337f74"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.116205 31359 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:09.116742 31374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.116884 31359 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:09.116955 31359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "67d15bb1aceb45f7b36518caf476a68e"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "3174ec9a6c93e7210c35f8fba8ce42b0"
server_key_iv: "7bde335d7839f42640e2afaf1f337f74"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.117020 31359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:09.134773 31359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:09.136010 31359 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:09.136126 31359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:09.136497 31359 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:09.136551 31359 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.136585 31359 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:09.136610 31359 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:09.143334 31359 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:33491
I20260201 03:08:09.143381 31489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:33491 every 8 connection(s)
I20260201 03:08:09.143637 31359 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:09.143785 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.136886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:39185 (local address 127.28.181.62:38061)
0201 03:08:09.136941 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.136944 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.137056 (+ 112us) server_negotiation.cc:408] Connection header received
0201 03:08:09.137387 (+ 331us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.137388 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.137399 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.137468 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.138132 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.138633 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.139113 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.139265 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.141144 (+ 1879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.141148 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.141148 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.141160 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.142532 (+ 1372us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.142889 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.142891 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.142891 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.142919 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.143112 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.143113 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.143114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.143216 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.143288 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.143575 (+ 287us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.143668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:09.143862 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.136827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.2:39185)
0201 03:08:09.136962 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.136987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.137316 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.137502 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.137504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.137764 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:09.138055 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.138057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.138731 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.138731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.139041 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.139042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.139160 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.139598 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:09.139627 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:09.141053 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:09.142635 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.142639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.142642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.142807 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.142975 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.142976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.142976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.143051 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.143278 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:09.143282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:09.143415 (+ 133us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.143541 (+ 126us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.143669 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260201 03:08:09.144641 31490 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38061
I20260201 03:08:09.144728 31490 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:09.144894 31490 heartbeater.cc:507] Master 127.28.181.62:38061 requested a full tablet report, sending...
I20260201 03:08:09.145294 31142 ts_manager.cc:194] Registered new tserver with Master: 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491)
I20260201 03:08:09.145644 31142 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:39185
I20260201 03:08:09.148854 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.145208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:49377 (local address 127.28.181.62:38061)
0201 03:08:09.145248 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.145249 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.145329 (+ 80us) server_negotiation.cc:408] Connection header received
0201 03:08:09.145352 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.145353 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.145360 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.145412 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.145720 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.146147 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.146565 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.146664 (+ 99us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.147116 (+ 452us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.147118 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.147118 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.147123 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.148229 (+ 1106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.148442 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.148443 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.148443 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.148458 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.148561 (+ 103us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.148562 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.148562 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.148633 (+ 71us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.148669 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.148756 (+ 87us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.148808 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260201 03:08:09.148856 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.145154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.2:49377)
0201 03:08:09.145268 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.145280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.145323 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.145429 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.145430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.145525 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:09.145654 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.145655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.146219 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.146220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.146505 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.146506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.146558 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.146867 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:09.146870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:09.147056 (+ 186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:09.148288 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.148289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.148289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.148392 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.148513 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.148513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.148514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.148531 (+ 17us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.148671 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:09.148671 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:09.148697 (+ 26us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.148742 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.148801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260201 03:08:09.151490 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31359
I20260201 03:08:09.151582 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:09.151729 29396 external_mini_cluster.cc:1468] Setting key 1b5ec6b046b9cd0b261fd2d182e4689a
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:09.186420 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38061
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:09.251626 31501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:09.251760 31501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:09.251778 31501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:09.252959 31501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:09.252995 31501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:09.253012 31501 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:08:09.253023 31501 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:08:09.253074 31501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:09.254487 31501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:38061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:09.254700 31501 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:09.254854 31501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:09.256825 31509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:09.256834 31507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:09.256847 31506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:09.256999 31501 server_base.cc:1060] running on GCE node
I20260201 03:08:09.257169 31501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:09.257350 31501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:09.258478 31501 hybrid_clock.cc:648] HybridClock initialized: now 1769915289258468 us; error 22 us; skew 500 ppm
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:09.260277 31501 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:09.260727 31501 webserver.cc:492] Webserver started at http://127.28.181.3:33279/ using document root <none> and password file <none>
I20260201 03:08:09.260926 31501 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:09.260969 31501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:09.261072 31501 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:09.261848 31501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "b0f62d70aebfe40ff5a8cb4f69d2b7ed"
server_key_iv: "022be66e421672f0b4da204264392e1a"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.262123 31501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "b0f62d70aebfe40ff5a8cb4f69d2b7ed"
server_key_iv: "022be66e421672f0b4da204264392e1a"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.263078 31501 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:09.263644 31516 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.263792 31501 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:09.263867 31501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
format_stamp: "Formatted at 2026-02-01 03:08:09 on dist-test-slave-ck1n"
server_key: "b0f62d70aebfe40ff5a8cb4f69d2b7ed"
server_key_iv: "022be66e421672f0b4da204264392e1a"
server_key_version: "encryptionkey@0"
I20260201 03:08:09.263942 31501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:09.272526 31501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:09.273869 31501 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:09.273996 31501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:09.274439 31501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:09.274483 31501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.274540 31501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:09.274564 31501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:09.280879 31501 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:40835
I20260201 03:08:09.280893 31631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:40835 every 8 connection(s)
I20260201 03:08:09.281232 31501 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:09.282202 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.274710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:57907 (local address 127.28.181.62:38061)
0201 03:08:09.274759 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.274761 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.274921 (+ 160us) server_negotiation.cc:408] Connection header received
0201 03:08:09.275434 (+ 513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.275434 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.275442 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.275480 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.276362 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.276800 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.277513 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.277678 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.279504 (+ 1826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.279509 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.279509 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.279516 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.281085 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.281449 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.281450 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.281451 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.281477 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.281687 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.281689 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.281689 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.281825 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.281897 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.282077 (+ 180us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.282147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260201 03:08:09.282251 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.274686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.3:57907)
0201 03:08:09.274813 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.274844 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.275353 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.275528 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.275531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.275921 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:09.276276 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.276279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.276884 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.276885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.277431 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.277434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.277562 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.278016 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:09.278037 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:09.279425 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:09.281162 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.281165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.281167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.281380 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.281544 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.281545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.281546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.281629 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.281888 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:09.281892 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:09.281975 (+ 83us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.282051 (+ 76us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.282140 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260201 03:08:09.282636 31632 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38061
I20260201 03:08:09.282699 31632 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:09.282881 31632 heartbeater.cc:507] Master 127.28.181.62:38061 requested a full tablet report, sending...
I20260201 03:08:09.283241 31143 ts_manager.cc:194] Registered new tserver with Master: 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:09.283710 31143 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:57907
I20260201 03:08:09.287313 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.283524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:41775 (local address 127.28.181.62:38061)
0201 03:08:09.283552 (+ 28us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.283554 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.283937 (+ 383us) server_negotiation.cc:408] Connection header received
0201 03:08:09.284007 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.284008 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.284017 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.284070 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.284363 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.284744 (+ 381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.285152 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.285237 (+ 85us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.285729 (+ 492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.285731 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.285731 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.285734 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.286699 (+ 965us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.286930 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.286930 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.286930 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.286945 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.287054 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.287054 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.287055 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.287107 (+ 52us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.287141 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.287225 (+ 84us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.287268 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260201 03:08:09.287338 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.283477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.28.181.3:41775)
0201 03:08:09.283852 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.283878 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.283936 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.284070 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.284070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.284168 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:09.284298 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.284299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.284803 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.284803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.285095 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.285096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.285143 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.285483 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:09.285486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:09.285673 (+ 187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:09.286759 (+ 1086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.286760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.286760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.286875 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.287003 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.287004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.287004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.287021 (+ 17us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.287144 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:09.287144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:09.287169 (+ 25us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.287207 (+ 38us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.287264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":66,"threads_started":1}
I20260201 03:08:09.289469 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31501
I20260201 03:08:09.289538 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:09.289645 29396 external_mini_cluster.cc:1468] Setting key 9adc075a8495ce25df82e16543f89dc7
I20260201 03:08:09.290352 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:09.307550 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.302442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.28.181.62:38061)
0201 03:08:09.302479 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.302480 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.302486 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:09.302527 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.302527 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.302533 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.302576 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.303088 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.303469 (+ 381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.303927 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.304015 (+ 88us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.305686 (+ 1671us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.305689 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.305689 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.305694 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.306707 (+ 1013us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.307005 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.307006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.307006 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.307030 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.307161 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.307161 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.307162 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.307228 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.307317 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.307403 (+ 86us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.307467 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:08:09.308707 31143 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:09.309741 31143 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:09.313978 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:40835 (local address 127.0.0.1:52776)
0201 03:08:09.311730 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.311740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.311801 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.311956 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.311958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.311960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.312178 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.312180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313022 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.313022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.313717 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.313719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313797 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.313828 (+ 31us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.313875 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.313916 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"spinlock_wait_cycles":896,"thread_start_us":74,"threads_started":1}
I20260201 03:08:09.314456 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.0.0.1:59264)
0201 03:08:09.311563 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.311587 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.311643 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.312021 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.312022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.312024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.312202 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.312204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313319 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.313320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.314247 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.314250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.314305 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.314312 (+ 7us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.314357 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.314392 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":42,"threads_started":1}
I20260201 03:08:09.314651 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.28.181.3:40835)
0201 03:08:09.311751 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.311757 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.311789 (+ 32us) server_negotiation.cc:408] Connection header received
0201 03:08:09.311852 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.311853 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.311863 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.312034 (+ 171us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.312253 (+ 219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.312932 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.313830 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.314458 (+ 628us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.314507 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.314547 (+ 40us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.314589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":56,"threads_started":1}
I20260201 03:08:09.314953 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33491 (local address 127.0.0.1:50182)
0201 03:08:09.311751 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.311761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.311809 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.312374 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.312375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.312377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.312596 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.312598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313631 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.313632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.314768 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.314770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.314838 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.314841 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.314866 (+ 25us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.314896 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":65,"threads_started":1}
I20260201 03:08:09.315053 31551 tablet_service.cc:1505] Processing CreateTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 (DEFAULT_TABLE table=test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.315238 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.28.181.1:37471)
0201 03:08:09.311903 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.311908 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.311914 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:09.311941 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.311942 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.311953 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.312019 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.312282 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313228 (+ 946us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.314598 (+ 1370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.315042 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.315084 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.315112 (+ 28us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.315146 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":66,"threads_started":1}
I20260201 03:08:09.315295 31551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b1752fb4d84d31961ef9c702a76c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.315618 31282 tablet_service.cc:1505] Processing CreateTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 (DEFAULT_TABLE table=test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.315685 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.311649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.28.181.2:33491)
0201 03:08:09.312151 (+ 502us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.312157 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.312164 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:09.312193 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.312195 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.312208 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.312261 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.312693 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.313534 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.315095 (+ 1561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.315515 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.315549 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.315578 (+ 29us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.315617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":66,"threads_started":1}
I20260201 03:08:09.315848 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b1752fb4d84d31961ef9c702a76c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.316118 31424 tablet_service.cc:1505] Processing CreateTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 (DEFAULT_TABLE table=test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.316325 31424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b1752fb4d84d31961ef9c702a76c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.317226 31652 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Bootstrap starting.
I20260201 03:08:09.317670 31652 tablet_bootstrap.cc:654] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.318200 31652 log.cc:826] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:09.318238 31653 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Bootstrap starting.
I20260201 03:08:09.318791 31655 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Bootstrap starting.
I20260201 03:08:09.318848 31653 tablet_bootstrap.cc:654] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.318835 31652 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: No bootstrap required, opened a new log
I20260201 03:08:09.318907 31652 ts_tablet_manager.cc:1403] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:09.319401 31653 log.cc:826] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:09.319588 31655 tablet_bootstrap.cc:654] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.320014 31653 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: No bootstrap required, opened a new log
I20260201 03:08:09.320070 31653 ts_tablet_manager.cc:1403] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:09.320148 31655 log.cc:826] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:09.320276 31652 raft_consensus.cc:359] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.320379 31652 raft_consensus.cc:385] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.320402 31652 raft_consensus.cc:740] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.320480 31652 consensus_queue.cc:260] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.320654 31655 tablet_bootstrap.cc:492] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: No bootstrap required, opened a new log
I20260201 03:08:09.320669 31652 ts_tablet_manager.cc:1434] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:09.320694 31632 heartbeater.cc:499] Master 127.28.181.62:38061 was elected leader, sending a full tablet report...
I20260201 03:08:09.320703 31655 ts_tablet_manager.cc:1403] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:09.321357 31653 raft_consensus.cc:359] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.321441 31653 raft_consensus.cc:385] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.321487 31653 raft_consensus.cc:740] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.321563 31653 consensus_queue.cc:260] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.321733 31653 ts_tablet_manager.cc:1434] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:09.321871 31348 heartbeater.cc:499] Master 127.28.181.62:38061 was elected leader, sending a full tablet report...
I20260201 03:08:09.322086 31655 raft_consensus.cc:359] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.322224 31655 raft_consensus.cc:385] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.322268 31655 raft_consensus.cc:740] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.322371 31655 consensus_queue.cc:260] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.322526 31655 ts_tablet_manager.cc:1434] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:09.322587 31490 heartbeater.cc:499] Master 127.28.181.62:38061 was elected leader, sending a full tablet report...
I20260201 03:08:09.362749 31658 raft_consensus.cc:493] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:09.362823 31658 raft_consensus.cc:515] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.363111 31658 leader_election.cc:290] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491)
I20260201 03:08:09.365079 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.363232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.28.181.3:40485)
0201 03:08:09.363286 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.363297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.363341 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.363500 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.363500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.363502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.363754 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.363755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364351 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.364352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.364873 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.364874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364923 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.364926 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.364969 (+ 43us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.365002 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260201 03:08:09.365104 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.363233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33491 (local address 127.28.181.3:46701)
0201 03:08:09.363437 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.363446 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.363496 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.363645 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.363646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.363647 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.363853 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.363854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364424 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.364424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.364894 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.364895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364973 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.364976 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.364998 (+ 22us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.365026 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":63,"thread_start_us":66,"threads_started":1}
I20260201 03:08:09.365413 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.363319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:40485 (local address 127.28.181.1:37471)
0201 03:08:09.363358 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.363360 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.363368 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:09.363405 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.363406 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.363415 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.363475 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.363826 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364278 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.364945 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.365328 (+ 383us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.365334 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.365353 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.365380 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:09.365515 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.363343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:46701 (local address 127.28.181.2:33491)
0201 03:08:09.363395 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.363397 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.363505 (+ 108us) server_negotiation.cc:408] Connection header received
0201 03:08:09.363554 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.363555 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.363565 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.363635 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.363922 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.364356 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.365008 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.365390 (+ 382us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.365396 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.365413 (+ 17us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.365432 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:09.365641 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0b1752fb4d84d31961ef9c702a76c95" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9" is_pre_election: true
I20260201 03:08:09.365739 31302 raft_consensus.cc:2468] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 0.
I20260201 03:08:09.365805 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0b1752fb4d84d31961ef9c702a76c95" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e" is_pre_election: true
I20260201 03:08:09.365918 31444 raft_consensus.cc:2468] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 0.
I20260201 03:08:09.365971 31518 leader_election.cc:304] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8700a7d3fbf741f5a68b01ca5ed043fb, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:09.366075 31658 raft_consensus.cc:2804] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:09.366127 31658 raft_consensus.cc:493] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:09.366158 31658 raft_consensus.cc:3060] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.366884 31658 raft_consensus.cc:515] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.367072 31658 leader_election.cc:290] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 election: Requested vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491)
I20260201 03:08:09.367169 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0b1752fb4d84d31961ef9c702a76c95" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e"
I20260201 03:08:09.367236 31444 raft_consensus.cc:3060] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.367866 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0b1752fb4d84d31961ef9c702a76c95" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9"
I20260201 03:08:09.367930 31302 raft_consensus.cc:3060] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.367945 31444 raft_consensus.cc:2468] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 1.
I20260201 03:08:09.368090 31517 leader_election.cc:304] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, 8700a7d3fbf741f5a68b01ca5ed043fb; no voters:
I20260201 03:08:09.368312 31658 raft_consensus.cc:2804] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:09.368463 31658 raft_consensus.cc:697] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 LEADER]: Becoming Leader. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Running, Role: LEADER
I20260201 03:08:09.368549 31658 consensus_queue.cc:237] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } }
I20260201 03:08:09.368631 31302 raft_consensus.cc:2468] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 1.
I20260201 03:08:09.369238 31141 catalog_manager.cc:5654] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3). New cstate: current_term: 1 leader_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } health_report { overall_health: UNKNOWN } } }
W20260201 03:08:09.394589 31491 tablet.cc:2378] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 01 03:08:09 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:09.407541 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.402409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38061 (local address 127.0.0.1:56486)
0201 03:08:09.402540 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.402554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.402626 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.402757 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.402758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.403035 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:09.403164 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.403166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.403692 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.403692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.404004 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.404005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.404051 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.404488 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:09.404492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:09.405611 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:09.406913 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.406917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.406919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.407064 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.407176 (+ 112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:09.407177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:09.407177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:09.407197 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:09.407323 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:09.407327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:09.407356 (+ 29us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.407404 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.407471 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260201 03:08:09.407621 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.402475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.28.181.62:38061)
0201 03:08:09.402527 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.402529 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.402603 (+ 74us) server_negotiation.cc:408] Connection header received
0201 03:08:09.402681 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.402682 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.402689 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.402740 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:09.403226 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.403625 (+ 399us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.404058 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.404147 (+ 89us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.405679 (+ 1532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:09.405683 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:09.405683 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:09.405688 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:09.406857 (+ 1169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.407116 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.407117 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.407117 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.407132 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:09.407230 (+ 98us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:09.407231 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:09.407231 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:09.407287 (+ 56us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:09.407322 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.407419 (+ 97us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.407506 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":28}
I20260201 03:08:09.409041 31141 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:56486:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:08:09.409214 31141 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:09.410400 31282 tablet_service.cc:1505] Processing CreateTablet for tablet 576cd281892a45769e47a584a3bff25d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cdb4a63084164223a53b5823fc90f7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:09.410406 31424 tablet_service.cc:1505] Processing CreateTablet for tablet 576cd281892a45769e47a584a3bff25d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cdb4a63084164223a53b5823fc90f7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:09.410480 31551 tablet_service.cc:1505] Processing CreateTablet for tablet 576cd281892a45769e47a584a3bff25d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cdb4a63084164223a53b5823fc90f7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:08:09.410512 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576cd281892a45769e47a584a3bff25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.410521 31424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576cd281892a45769e47a584a3bff25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.410566 31551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576cd281892a45769e47a584a3bff25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.411321 31653 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: Bootstrap starting.
I20260201 03:08:09.411621 31652 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: Bootstrap starting.
I20260201 03:08:09.411643 31655 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: Bootstrap starting.
I20260201 03:08:09.411702 31653 tablet_bootstrap.cc:654] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.412034 31652 tablet_bootstrap.cc:654] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.412057 31655 tablet_bootstrap.cc:654] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.412391 31653 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: No bootstrap required, opened a new log
I20260201 03:08:09.412431 31653 ts_tablet_manager.cc:1403] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:09.412556 31653 raft_consensus.cc:359] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.412621 31653 raft_consensus.cc:385] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.412644 31653 raft_consensus.cc:740] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.412685 31653 consensus_queue.cc:260] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.412719 31655 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: No bootstrap required, opened a new log
I20260201 03:08:09.412712 31652 tablet_bootstrap.cc:492] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: No bootstrap required, opened a new log
I20260201 03:08:09.412751 31652 ts_tablet_manager.cc:1403] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:09.412751 31655 ts_tablet_manager.cc:1403] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:09.412760 31653 ts_tablet_manager.cc:1434] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.412791 31659 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.412837 31659 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.412871 31652 raft_consensus.cc:359] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.412881 31655 raft_consensus.cc:359] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.412923 31652 raft_consensus.cc:385] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.412936 31655 raft_consensus.cc:385] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.412942 31652 raft_consensus.cc:740] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.412957 31655 raft_consensus.cc:740] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.412987 31652 consensus_queue.cc:260] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.412997 31655 consensus_queue.cc:260] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.413056 31652 ts_tablet_manager.cc:1434] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:09.413059 31655 ts_tablet_manager.cc:1434] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.413157 31662 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.413164 31660 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.413206 31660 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.413206 31662 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.486012 31659 raft_consensus.cc:493] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:09.486088 31659 raft_consensus.cc:515] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.486340 31659 leader_election.cc:290] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:09.488648 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.486444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33491 (local address 127.28.181.1:36745)
0201 03:08:09.486493 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.486502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.486541 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.486709 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.486710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.486711 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.486893 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.486894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.487640 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.487641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.488463 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.488465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.488525 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.488528 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.488557 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.488584 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260201 03:08:09.488960 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.486567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:36745 (local address 127.28.181.2:33491)
0201 03:08:09.486601 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.486603 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.486611 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:09.486632 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.486633 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.486641 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.486696 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.486962 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.487563 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.488539 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.488848 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.488870 (+ 22us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.488888 (+ 18us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.488910 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260201 03:08:09.488981 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.486466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:40835 (local address 127.28.181.1:51129)
0201 03:08:09.486567 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.486574 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.486620 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.486709 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.486710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.486711 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:09.486904 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.486905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.487640 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.487641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.488820 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.488822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.488897 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.488899 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.488920 (+ 21us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.488942 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260201 03:08:09.489101 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576cd281892a45769e47a584a3bff25d" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e" is_pre_election: true
I20260201 03:08:09.489166 31444 raft_consensus.cc:2468] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 0.
I20260201 03:08:09.489311 31233 leader_election.cc:304] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:09.489410 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.486572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:51129 (local address 127.28.181.3:40835)
0201 03:08:09.486606 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.486608 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.486615 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:09.486656 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.486657 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.486663 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.486709 (+ 46us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:09.486963 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.487574 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.488893 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.489313 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.489319 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.489335 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.489355 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260201 03:08:09.489431 31659 raft_consensus.cc:2804] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:09.489473 31659 raft_consensus.cc:493] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:09.489496 31659 raft_consensus.cc:3060] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.489610 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576cd281892a45769e47a584a3bff25d" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" is_pre_election: true
I20260201 03:08:09.489673 31586 raft_consensus.cc:2468] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 0.
I20260201 03:08:09.490151 31659 raft_consensus.cc:515] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.490299 31659 leader_election.cc:290] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 election: Requested vote from peers 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:09.490411 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576cd281892a45769e47a584a3bff25d" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e"
I20260201 03:08:09.490428 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576cd281892a45769e47a584a3bff25d" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
I20260201 03:08:09.490475 31586 raft_consensus.cc:3060] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.490475 31444 raft_consensus.cc:3060] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.491185 31586 raft_consensus.cc:2468] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 1.
I20260201 03:08:09.491369 31235 leader_election.cc:304] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8700a7d3fbf741f5a68b01ca5ed043fb, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:09.491441 31444 raft_consensus.cc:2468] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 1.
I20260201 03:08:09.491449 31659 raft_consensus.cc:2804] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:09.491593 31659 raft_consensus.cc:697] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [term 1 LEADER]: Becoming Leader. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Running, Role: LEADER
I20260201 03:08:09.491669 31659 consensus_queue.cc:237] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.492048 31674 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: TxnStatusTablet state changed. Reason: New leader a25056fed19b44f2902a3462ba9d17d9. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.492117 31674 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:09.492255 31676 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:08:09.492278 31141 catalog_manager.cc:5654] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:09.492358 31676 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260201 03:08:09.494696 31349 tablet.cc:2378] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:09.504946 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.503202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.28.181.1:37471)
0201 03:08:09.503236 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.503238 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.503244 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:09.503310 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.503311 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.503323 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.503375 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.503570 (+ 195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.503968 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.504472 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.504613 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.504636 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.504698 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.504785 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.504866 (+ 81us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.504890 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:09.504954 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.503141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.0.0.1:59270)
0201 03:08:09.503201 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.503209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.503248 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.503382 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.503383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.503383 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:09.503509 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.503510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.504042 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.504043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.504400 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.504401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.504475 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.504490 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.504786 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.504796 (+ 10us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.504851 (+ 55us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.504896 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260201 03:08:09.506247 31586 raft_consensus.cc:1275] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Refusing update from remote peer a25056fed19b44f2902a3462ba9d17d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.506274 31444 raft_consensus.cc:1275] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Refusing update from remote peer a25056fed19b44f2902a3462ba9d17d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.506466 31659 consensus_queue.cc:1048] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:09.506556 31674 consensus_queue.cc:1048] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:09.507419 31662 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: TxnStatusTablet state changed. Reason: New leader a25056fed19b44f2902a3462ba9d17d9. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.507493 31662 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.507544 31660 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: TxnStatusTablet state changed. Reason: New leader a25056fed19b44f2902a3462ba9d17d9. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.507616 31660 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.508157 31674 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.508162 31679 mvcc.cc:204] Tried to move back new op lower bound from 7249573025815502848 to 7249573025758433280. Current Snapshot: MvccSnapshot[applied={T|T < 7249573025815502848}]
I20260201 03:08:09.508239 31674 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:09.508347 31662 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.508415 31662 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 8700a7d3fbf741f5a68b01ca5ed043fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.508730 31680 mvcc.cc:204] Tried to move back new op lower bound from 7249573025815502848 to 7249573025758433280. Current Snapshot: MvccSnapshot[applied={T|T < 7249573025815502848 or (T in {7249573025815502848})}]
I20260201 03:08:09.509220 31674 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.509279 31674 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P a25056fed19b44f2902a3462ba9d17d9: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:08:09.509714 31660 tablet_replica.cc:442] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } } }
I20260201 03:08:09.509799 31660 tablet_replica.cc:445] T 576cd281892a45769e47a584a3bff25d P 67d15bb1aceb45f7b36518caf476a68e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:08:09.512138 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.510394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.28.181.3:40835)
0201 03:08:09.510430 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.510431 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.510439 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:09.510499 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.510500 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.510510 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.510549 (+ 39us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.510794 (+ 245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.511140 (+ 346us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.511807 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.511897 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.511917 (+ 20us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.511975 (+ 58us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.512020 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.512087 (+ 67us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.512104 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:09.515508 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.513972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:55791 (local address 127.28.181.1:37471)
0201 03:08:09.514001 (+ 29us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.514003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.514026 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:08:09.514062 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.514063 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.514069 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.514129 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.514326 (+ 197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.514697 (+ 371us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.515173 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.515293 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.515306 (+ 13us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.515320 (+ 14us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.515371 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.515442 (+ 71us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.515458 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:08:09.515524 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.513927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.28.181.3:55791)
0201 03:08:09.513975 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.513983 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.514027 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.514133 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.514134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.514135 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:09.514263 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.514264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.514774 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.514775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.515098 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.515099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.515163 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.515176 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.515384 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.515386 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.515440 (+ 54us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.515473 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260201 03:08:09.517565 31444 raft_consensus.cc:1275] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Refusing update from remote peer 8700a7d3fbf741f5a68b01ca5ed043fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.517622 31302 raft_consensus.cc:1275] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Refusing update from remote peer 8700a7d3fbf741f5a68b01ca5ed043fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.517812 31662 consensus_queue.cc:1048] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:09.517881 31658 consensus_queue.cc:1048] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:09.519343 31681 mvcc.cc:204] Tried to move back new op lower bound from 7249573025862672384 to 7249573025254260736. Current Snapshot: MvccSnapshot[applied={T|T < 7249573025862672384}]
I20260201 03:08:09.531719 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.522873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:47969 (local address 127.28.181.3:40835)
0201 03:08:09.522916 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.522918 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.529901 (+ 6983us) server_negotiation.cc:408] Connection header received
0201 03:08:09.529952 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.529953 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.529961 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.530007 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.530292 (+ 285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.530897 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.531357 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.531497 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.531513 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.531533 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.531585 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.531650 (+ 65us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.531666 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260201 03:08:09.531746 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.522829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:40835 (local address 127.28.181.1:47969)
0201 03:08:09.529841 (+ 7012us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.529851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.529906 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.530045 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.530046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.530048 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:09.530228 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.530229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.530962 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.530962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.531294 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.531295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.531341 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.531351 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.531596 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.531597 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.531634 (+ 37us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.531684 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6933,"thread_start_us":76,"threads_started":1}
I20260201 03:08:09.538452 31143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:09.538517 31143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:09.539098 31143 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Sending DeleteTablet for 3 replicas of tablet d0b1752fb4d84d31961ef9c702a76c95
I20260201 03:08:09.539336 31551 tablet_service.cc:1552] Processing DeleteTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:09 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:52776
I20260201 03:08:09.539369 31424 tablet_service.cc:1552] Processing DeleteTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:09 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260201 03:08:09.539541 31699 tablet_replica.cc:333] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: stopping tablet replica
I20260201 03:08:09.539603 31699 raft_consensus.cc:2243] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:09.539662 31699 raft_consensus.cc:2272] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:09.539680 31282 tablet_service.cc:1552] Processing DeleteTablet for tablet d0b1752fb4d84d31961ef9c702a76c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:09 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260201 03:08:09.540066 31143 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:09.540169 31143 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:09.541172 31551 tablet_service.cc:1505] Processing CreateTablet for tablet 44dcc6a764c0445895569f15ea72d8fe (DEFAULT_TABLE table=test-table [id=abbf17af11da4708a523bff9267bf7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.541191 31424 tablet_service.cc:1505] Processing CreateTablet for tablet 44dcc6a764c0445895569f15ea72d8fe (DEFAULT_TABLE table=test-table [id=abbf17af11da4708a523bff9267bf7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.541262 31551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44dcc6a764c0445895569f15ea72d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.541281 31424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44dcc6a764c0445895569f15ea72d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.542318 31652 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Bootstrap starting.
I20260201 03:08:09.542372 31655 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Bootstrap starting.
I20260201 03:08:09.542707 31652 tablet_bootstrap.cc:654] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.542761 31655 tablet_bootstrap.cc:654] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.542860 31282 tablet_service.cc:1505] Processing CreateTablet for tablet 44dcc6a764c0445895569f15ea72d8fe (DEFAULT_TABLE table=test-table [id=abbf17af11da4708a523bff9267bf7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:09.542948 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44dcc6a764c0445895569f15ea72d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:09.544457 31653 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Bootstrap starting.
I20260201 03:08:09.544795 31699 ts_tablet_manager.cc:1916] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:09.544849 31653 tablet_bootstrap.cc:654] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:09.545568 31700 tablet_replica.cc:333] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: stopping tablet replica
I20260201 03:08:09.545644 31700 raft_consensus.cc:2243] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:09.545691 31700 raft_consensus.cc:2272] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:09.545753 31653 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: No bootstrap required, opened a new log
I20260201 03:08:09.545822 31653 ts_tablet_manager.cc:1403] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:09.545955 31653 raft_consensus.cc:359] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.546023 31653 raft_consensus.cc:385] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.546056 31653 raft_consensus.cc:740] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.546092 31653 consensus_queue.cc:260] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.546171 31653 ts_tablet_manager.cc:1434] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.546638 31655 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: No bootstrap required, opened a new log
I20260201 03:08:09.546679 31655 ts_tablet_manager.cc:1403] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260201 03:08:09.546810 31655 raft_consensus.cc:359] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.546868 31655 raft_consensus.cc:385] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.546886 31655 raft_consensus.cc:740] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.546924 31655 consensus_queue.cc:260] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.546998 31655 ts_tablet_manager.cc:1434] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.547792 31699 ts_tablet_manager.cc:1929] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:09.547895 31699 log.cc:1199] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/d0b1752fb4d84d31961ef9c702a76c95
I20260201 03:08:09.548175 31699 ts_tablet_manager.cc:1950] T d0b1752fb4d84d31961ef9c702a76c95 P 67d15bb1aceb45f7b36518caf476a68e: Deleting consensus metadata
I20260201 03:08:09.548507 31698 tablet_replica.cc:333] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: stopping tablet replica
I20260201 03:08:09.548543 31127 catalog_manager.cc:4985] TS 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491): tablet d0b1752fb4d84d31961ef9c702a76c95 (table test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]) successfully deleted
I20260201 03:08:09.546029 31700 ts_tablet_manager.cc:1916] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:09.548652 31698 raft_consensus.cc:2243] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:09.548775 31698 raft_consensus.cc:2272] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:09.549134 31698 ts_tablet_manager.cc:1916] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:09.549955 31700 ts_tablet_manager.cc:1929] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:09.550016 31700 log.cc:1199] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/d0b1752fb4d84d31961ef9c702a76c95
I20260201 03:08:09.550237 31700 ts_tablet_manager.cc:1950] T d0b1752fb4d84d31961ef9c702a76c95 P a25056fed19b44f2902a3462ba9d17d9: Deleting consensus metadata
I20260201 03:08:09.550549 31128 catalog_manager.cc:4985] TS a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471): tablet d0b1752fb4d84d31961ef9c702a76c95 (table test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]) successfully deleted
I20260201 03:08:09.550765 31652 tablet_bootstrap.cc:492] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: No bootstrap required, opened a new log
I20260201 03:08:09.550830 31652 ts_tablet_manager.cc:1403] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260201 03:08:09.550951 31652 raft_consensus.cc:359] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.551004 31652 raft_consensus.cc:385] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:09.551021 31652 raft_consensus.cc:740] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Initialized, Role: FOLLOWER
I20260201 03:08:09.551064 31652 consensus_queue.cc:260] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.551153 31652 ts_tablet_manager.cc:1434] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:09.551251 31698 ts_tablet_manager.cc:1929] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:09.551384 31698 log.cc:1199] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/d0b1752fb4d84d31961ef9c702a76c95
I20260201 03:08:09.551798 31698 ts_tablet_manager.cc:1950] T d0b1752fb4d84d31961ef9c702a76c95 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting consensus metadata
I20260201 03:08:09.552227 31129 catalog_manager.cc:4985] TS 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835): tablet d0b1752fb4d84d31961ef9c702a76c95 (table test-table [id=003d7174df4a4a8ab1f59b8fe1c6ad03]) successfully deleted
I20260201 03:08:09.732365 31674 raft_consensus.cc:493] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:09.732450 31674 raft_consensus.cc:515] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.732607 31674 leader_election.cc:290] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:09.732758 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44dcc6a764c0445895569f15ea72d8fe" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e" is_pre_election: true
I20260201 03:08:09.732856 31444 raft_consensus.cc:2468] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 0.
I20260201 03:08:09.732833 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44dcc6a764c0445895569f15ea72d8fe" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" is_pre_election: true
I20260201 03:08:09.732924 31586 raft_consensus.cc:2468] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 0.
I20260201 03:08:09.733000 31233 leader_election.cc:304] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:09.733083 31674 raft_consensus.cc:2804] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:09.733135 31674 raft_consensus.cc:493] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:09.733158 31674 raft_consensus.cc:3060] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.733841 31674 raft_consensus.cc:515] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.733973 31674 leader_election.cc:290] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 election: Requested vote from peers 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:09.734117 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44dcc6a764c0445895569f15ea72d8fe" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e"
I20260201 03:08:09.734174 31444 raft_consensus.cc:3060] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.734127 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44dcc6a764c0445895569f15ea72d8fe" candidate_uuid: "a25056fed19b44f2902a3462ba9d17d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
I20260201 03:08:09.734198 31586 raft_consensus.cc:3060] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:09.734721 31444 raft_consensus.cc:2468] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 1.
I20260201 03:08:09.734741 31586 raft_consensus.cc:2468] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25056fed19b44f2902a3462ba9d17d9 in term 1.
I20260201 03:08:09.734880 31233 leader_election.cc:304] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:09.735005 31674 raft_consensus.cc:2804] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:09.735057 31674 raft_consensus.cc:697] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 1 LEADER]: Becoming Leader. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Running, Role: LEADER
I20260201 03:08:09.735105 31674 consensus_queue.cc:237] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:09.735617 31143 catalog_manager.cc:5654] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1). New cstate: current_term: 1 leader_uuid: "a25056fed19b44f2902a3462ba9d17d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:09.772464 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.771109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.28.181.1:37471)
0201 03:08:09.771163 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.771165 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.771172 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:09.771188 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.771188 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.771195 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.771238 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.771421 (+ 183us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.771785 (+ 364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.772199 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.772286 (+ 87us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.772299 (+ 13us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.772314 (+ 15us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.772350 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.772407 (+ 57us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.772422 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:09.775516 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.774065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.28.181.1:60925)
0201 03:08:09.774119 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:09.774139 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:09.774175 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:09.774284 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:09.774284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:09.774285 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:09.774389 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.774390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.774874 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.774874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:09.775184 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:09.775185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.775232 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.775234 (+ 2us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.775404 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.775405 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:09.775436 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:08:09.775463 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260201 03:08:09.775517 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:09.774148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:60925 (local address 127.28.181.1:37471)
0201 03:08:09.774191 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:09.774193 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:09.774199 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:09.774222 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:09.774222 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:09.774228 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:09.774270 (+ 42us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:09.774437 (+ 167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.774803 (+ 366us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:09.775238 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:09.775325 (+ 87us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:09.775337 (+ 12us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:09.775350 (+ 13us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:09.775387 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:09.775450 (+ 63us) server_negotiation.cc:300] Negotiation successful
0201 03:08:09.775464 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260201 03:08:09.776989 31444 raft_consensus.cc:1275] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Refusing update from remote peer a25056fed19b44f2902a3462ba9d17d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.777006 31586 raft_consensus.cc:1275] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Refusing update from remote peer a25056fed19b44f2902a3462ba9d17d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:09.777172 31659 consensus_queue.cc:1048] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:09.777235 31686 consensus_queue.cc:1048] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260201 03:08:09.780867 31679 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260201 03:08:11.298029 31141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:11.298113 31141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:11.298857 31141 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Sending DeleteTablet for 3 replicas of tablet 44dcc6a764c0445895569f15ea72d8fe
I20260201 03:08:11.299098 31551 tablet_service.cc:1552] Processing DeleteTablet for tablet 44dcc6a764c0445895569f15ea72d8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:52776
I20260201 03:08:11.299098 31424 tablet_service.cc:1552] Processing DeleteTablet for tablet 44dcc6a764c0445895569f15ea72d8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260201 03:08:11.299145 31282 tablet_service.cc:1552] Processing DeleteTablet for tablet 44dcc6a764c0445895569f15ea72d8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260201 03:08:11.300189 31730 tablet_replica.cc:333] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: stopping tablet replica
I20260201 03:08:11.300300 31730 raft_consensus.cc:2243] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:11.300390 31730 raft_consensus.cc:2272] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.300513 31729 tablet_replica.cc:333] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: stopping tablet replica
I20260201 03:08:11.300591 31729 raft_consensus.cc:2243] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:11.300662 31729 raft_consensus.cc:2272] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.300678 31730 ts_tablet_manager.cc:1916] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.300904 31729 ts_tablet_manager.cc:1916] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.301856 31731 tablet_replica.cc:333] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: stopping tablet replica
I20260201 03:08:11.301921 31731 raft_consensus.cc:2243] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:11.302017 31731 raft_consensus.cc:2272] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.302194 31730 ts_tablet_manager.cc:1929] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:08:11.302273 31731 ts_tablet_manager.cc:1916] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.302299 31730 log.cc:1199] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/44dcc6a764c0445895569f15ea72d8fe
I20260201 03:08:11.302388 31729 ts_tablet_manager.cc:1929] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:08:11.302435 31729 log.cc:1199] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/44dcc6a764c0445895569f15ea72d8fe
I20260201 03:08:11.302651 31730 ts_tablet_manager.cc:1950] T 44dcc6a764c0445895569f15ea72d8fe P 67d15bb1aceb45f7b36518caf476a68e: Deleting consensus metadata
I20260201 03:08:11.302722 31729 ts_tablet_manager.cc:1950] T 44dcc6a764c0445895569f15ea72d8fe P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting consensus metadata
I20260201 03:08:11.302978 31127 catalog_manager.cc:4985] TS 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491): tablet 44dcc6a764c0445895569f15ea72d8fe (table test-table [id=abbf17af11da4708a523bff9267bf7eb]) successfully deleted
I20260201 03:08:11.303002 31129 catalog_manager.cc:4985] TS 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835): tablet 44dcc6a764c0445895569f15ea72d8fe (table test-table [id=abbf17af11da4708a523bff9267bf7eb]) successfully deleted
I20260201 03:08:11.303719 31731 ts_tablet_manager.cc:1929] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:08:11.303790 31731 log.cc:1199] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/44dcc6a764c0445895569f15ea72d8fe
I20260201 03:08:11.303988 31731 ts_tablet_manager.cc:1950] T 44dcc6a764c0445895569f15ea72d8fe P a25056fed19b44f2902a3462ba9d17d9: Deleting consensus metadata
I20260201 03:08:11.304286 31128 catalog_manager.cc:4985] TS a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471): tablet 44dcc6a764c0445895569f15ea72d8fe (table test-table [id=abbf17af11da4708a523bff9267bf7eb]) successfully deleted
Feb 01 03:08:11 dist-test-slave-ck1n krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:08:11 dist-test-slave-ck1n krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:11.318614 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.313284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.28.181.62:38061)
0201 03:08:11.313428 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.313430 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.313439 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:11.313457 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.313458 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.313466 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.313513 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:11.314144 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.314594 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.315046 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.315141 (+ 95us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.316739 (+ 1598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:11.316742 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:11.316743 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:11.316748 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:11.317994 (+ 1246us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:11.318208 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:11.318209 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:11.318209 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:11.318222 (+ 13us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:11.318318 (+ 96us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:11.318318 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:11.318318 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:11.318380 (+ 62us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:11.318438 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.318508 (+ 70us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.318560 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260201 03:08:11.319803 31141 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:11.319944 31141 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:11.321100 31282 tablet_service.cc:1505] Processing CreateTablet for tablet 192dba8400e940959bc14aa59785d1b8 (DEFAULT_TABLE table=test-table [id=b5755463788349668ea5b0e9cadef381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.321099 31424 tablet_service.cc:1505] Processing CreateTablet for tablet 192dba8400e940959bc14aa59785d1b8 (DEFAULT_TABLE table=test-table [id=b5755463788349668ea5b0e9cadef381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.321120 31551 tablet_service.cc:1505] Processing CreateTablet for tablet 192dba8400e940959bc14aa59785d1b8 (DEFAULT_TABLE table=test-table [id=b5755463788349668ea5b0e9cadef381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.321198 31424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192dba8400e940959bc14aa59785d1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.321198 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192dba8400e940959bc14aa59785d1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.321223 31551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192dba8400e940959bc14aa59785d1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.322288 31742 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Bootstrap starting.
I20260201 03:08:11.322343 31740 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Bootstrap starting.
I20260201 03:08:11.322665 31742 tablet_bootstrap.cc:654] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.322731 31740 tablet_bootstrap.cc:654] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.322979 31741 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Bootstrap starting.
I20260201 03:08:11.323199 31742 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: No bootstrap required, opened a new log
I20260201 03:08:11.323248 31742 ts_tablet_manager.cc:1403] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.323278 31740 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: No bootstrap required, opened a new log
I20260201 03:08:11.323318 31740 ts_tablet_manager.cc:1403] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.323320 31741 tablet_bootstrap.cc:654] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.323407 31742 raft_consensus.cc:359] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.323428 31740 raft_consensus.cc:359] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.323482 31742 raft_consensus.cc:385] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.323495 31740 raft_consensus.cc:385] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.323504 31742 raft_consensus.cc:740] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.323518 31740 raft_consensus.cc:740] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.323562 31742 consensus_queue.cc:260] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.323593 31740 consensus_queue.cc:260] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.323669 31742 ts_tablet_manager.cc:1434] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:11.323786 31740 ts_tablet_manager.cc:1434] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:11.324142 31741 tablet_bootstrap.cc:492] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: No bootstrap required, opened a new log
I20260201 03:08:11.324218 31741 ts_tablet_manager.cc:1403] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.324353 31741 raft_consensus.cc:359] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.324421 31741 raft_consensus.cc:385] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.324445 31741 raft_consensus.cc:740] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.324482 31741 consensus_queue.cc:260] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.324609 31741 ts_tablet_manager.cc:1434] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260201 03:08:11.533247 31633 tablet.cc:2378] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:11.606038 31746 raft_consensus.cc:493] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:11.606106 31746 raft_consensus.cc:515] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.606231 31746 leader_election.cc:290] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491)
I20260201 03:08:11.606400 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "192dba8400e940959bc14aa59785d1b8" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e" is_pre_election: true
I20260201 03:08:11.606397 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "192dba8400e940959bc14aa59785d1b8" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9" is_pre_election: true
I20260201 03:08:11.606465 31444 raft_consensus.cc:2468] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 0.
I20260201 03:08:11.606489 31302 raft_consensus.cc:2468] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 0.
I20260201 03:08:11.606604 31517 leader_election.cc:304] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, 8700a7d3fbf741f5a68b01ca5ed043fb; no voters:
I20260201 03:08:11.606710 31746 raft_consensus.cc:2804] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:11.606770 31746 raft_consensus.cc:493] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:11.606796 31746 raft_consensus.cc:3060] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.607385 31746 raft_consensus.cc:515] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.607532 31746 leader_election.cc:290] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 election: Requested vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491)
I20260201 03:08:11.607668 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "192dba8400e940959bc14aa59785d1b8" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9"
I20260201 03:08:11.607679 31444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "192dba8400e940959bc14aa59785d1b8" candidate_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d15bb1aceb45f7b36518caf476a68e"
I20260201 03:08:11.607736 31444 raft_consensus.cc:3060] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.607736 31302 raft_consensus.cc:3060] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.608397 31444 raft_consensus.cc:2468] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 1.
I20260201 03:08:11.608446 31302 raft_consensus.cc:2468] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8700a7d3fbf741f5a68b01ca5ed043fb in term 1.
I20260201 03:08:11.608565 31517 leader_election.cc:304] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, 8700a7d3fbf741f5a68b01ca5ed043fb; no voters:
I20260201 03:08:11.608666 31746 raft_consensus.cc:2804] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:11.608832 31746 raft_consensus.cc:697] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 LEADER]: Becoming Leader. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Running, Role: LEADER
I20260201 03:08:11.608878 31746 consensus_queue.cc:237] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.609401 31143 catalog_manager.cc:5654] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3). New cstate: current_term: 1 leader_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } health_report { overall_health: HEALTHY } } }
I20260201 03:08:11.677011 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.675264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.28.181.3:40835)
0201 03:08:11.675411 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.675413 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.675420 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:11.675435 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.675436 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.675443 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.675503 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:11.675727 (+ 224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.676132 (+ 405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.676630 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.676759 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.676774 (+ 15us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.676792 (+ 18us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.676851 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.676921 (+ 70us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.676942 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260201 03:08:11.678849 31444 raft_consensus.cc:1275] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Refusing update from remote peer 8700a7d3fbf741f5a68b01ca5ed043fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:11.678866 31302 raft_consensus.cc:1275] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Refusing update from remote peer 8700a7d3fbf741f5a68b01ca5ed043fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:11.679050 31748 consensus_queue.cc:1048] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:11.679114 31746 consensus_queue.cc:1048] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:11.688153 31143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:11.688223 31143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:11.688731 31143 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Sending DeleteTablet for 3 replicas of tablet 192dba8400e940959bc14aa59785d1b8
I20260201 03:08:11.688964 31424 tablet_service.cc:1552] Processing DeleteTablet for tablet 192dba8400e940959bc14aa59785d1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260201 03:08:11.688988 31551 tablet_service.cc:1552] Processing DeleteTablet for tablet 192dba8400e940959bc14aa59785d1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:52776
I20260201 03:08:11.689009 31282 tablet_service.cc:1552] Processing DeleteTablet for tablet 192dba8400e940959bc14aa59785d1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:11 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260201 03:08:11.689073 31730 tablet_replica.cc:333] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: stopping tablet replica
I20260201 03:08:11.689100 31731 tablet_replica.cc:333] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: stopping tablet replica
I20260201 03:08:11.689121 31730 raft_consensus.cc:2243] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:11.689147 31731 raft_consensus.cc:2243] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:11.689167 31730 raft_consensus.cc:2272] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.689186 31731 raft_consensus.cc:2272] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.689229 31729 tablet_replica.cc:333] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: stopping tablet replica
I20260201 03:08:11.689271 31729 raft_consensus.cc:2243] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:11.689311 31729 raft_consensus.cc:2272] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:11.689257 31143 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:11.689368 31143 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:11.689584 31730 ts_tablet_manager.cc:1916] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.689743 31731 ts_tablet_manager.cc:1916] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.690865 31730 ts_tablet_manager.cc:1929] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:11.690912 31730 log.cc:1199] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/192dba8400e940959bc14aa59785d1b8
I20260201 03:08:11.690914 31729 ts_tablet_manager.cc:1916] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:11.691120 31730 ts_tablet_manager.cc:1950] T 192dba8400e940959bc14aa59785d1b8 P 67d15bb1aceb45f7b36518caf476a68e: Deleting consensus metadata
I20260201 03:08:11.691430 31127 catalog_manager.cc:4985] TS 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491): tablet 192dba8400e940959bc14aa59785d1b8 (table test-table [id=b5755463788349668ea5b0e9cadef381]) successfully deleted
I20260201 03:08:11.691877 31282 tablet_service.cc:1505] Processing CreateTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 (DEFAULT_TABLE table=test-table [id=b59fa6612fb044b49636fd6f672906f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.691877 31424 tablet_service.cc:1505] Processing CreateTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 (DEFAULT_TABLE table=test-table [id=b59fa6612fb044b49636fd6f672906f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.691963 31424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135dac9cd2924c4298c4ee2c5b0feba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.692220 31729 ts_tablet_manager.cc:1929] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:11.692253 31729 log.cc:1199] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/192dba8400e940959bc14aa59785d1b8
I20260201 03:08:11.692378 31729 ts_tablet_manager.cc:1950] T 192dba8400e940959bc14aa59785d1b8 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting consensus metadata
I20260201 03:08:11.692464 31551 tablet_service.cc:1505] Processing CreateTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 (DEFAULT_TABLE table=test-table [id=b59fa6612fb044b49636fd6f672906f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:11.692510 31551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135dac9cd2924c4298c4ee2c5b0feba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.692731 31129 catalog_manager.cc:4985] TS 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835): tablet 192dba8400e940959bc14aa59785d1b8 (table test-table [id=b5755463788349668ea5b0e9cadef381]) successfully deleted
I20260201 03:08:11.692793 31741 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Bootstrap starting.
I20260201 03:08:11.693079 31741 tablet_bootstrap.cc:654] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.693526 31740 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Bootstrap starting.
I20260201 03:08:11.693648 31741 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: No bootstrap required, opened a new log
I20260201 03:08:11.693691 31741 ts_tablet_manager.cc:1403] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.693806 31731 ts_tablet_manager.cc:1929] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:08:11.693822 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135dac9cd2924c4298c4ee2c5b0feba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:11.693867 31731 log.cc:1199] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/192dba8400e940959bc14aa59785d1b8
I20260201 03:08:11.693871 31740 tablet_bootstrap.cc:654] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.693991 31741 raft_consensus.cc:359] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.694052 31741 raft_consensus.cc:385] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.694057 31731 ts_tablet_manager.cc:1950] T 192dba8400e940959bc14aa59785d1b8 P a25056fed19b44f2902a3462ba9d17d9: Deleting consensus metadata
I20260201 03:08:11.694072 31741 raft_consensus.cc:740] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.694133 31741 consensus_queue.cc:260] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.694217 31741 ts_tablet_manager.cc:1434] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:11.694437 31128 catalog_manager.cc:4985] TS a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471): tablet 192dba8400e940959bc14aa59785d1b8 (table test-table [id=b5755463788349668ea5b0e9cadef381]) successfully deleted
I20260201 03:08:11.694586 31740 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: No bootstrap required, opened a new log
I20260201 03:08:11.694626 31740 ts_tablet_manager.cc:1403] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.694739 31740 raft_consensus.cc:359] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.694813 31740 raft_consensus.cc:385] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.694947 31740 raft_consensus.cc:740] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8700a7d3fbf741f5a68b01ca5ed043fb, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.695051 31742 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9: Bootstrap starting.
I20260201 03:08:11.695066 31740 consensus_queue.cc:260] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.695145 31740 ts_tablet_manager.cc:1434] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:11.695343 31742 tablet_bootstrap.cc:654] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:11.695896 31742 tablet_bootstrap.cc:492] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9: No bootstrap required, opened a new log
I20260201 03:08:11.695937 31742 ts_tablet_manager.cc:1403] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:11.696024 31742 raft_consensus.cc:359] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.696062 31742 raft_consensus.cc:385] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:11.696074 31742 raft_consensus.cc:740] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25056fed19b44f2902a3462ba9d17d9, State: Initialized, Role: FOLLOWER
I20260201 03:08:11.696118 31742 consensus_queue.cc:260] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.696187 31742 ts_tablet_manager.cc:1434] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:11.952648 31747 raft_consensus.cc:493] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:11.952750 31747 raft_consensus.cc:515] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.953097 31747 leader_election.cc:290] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:11.955297 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.953284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.28.181.2:43579)
0201 03:08:11.953472 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:11.953483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:11.953559 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:11.953732 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:11.953733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:11.953735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:11.953975 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.953977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.954585 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.954585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:11.955079 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.955080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.955157 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.955161 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:11.955188 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:11.955219 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":12,"thread_start_us":72,"threads_started":1}
I20260201 03:08:11.955400 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.953284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:40835 (local address 127.28.181.2:44469)
0201 03:08:11.953409 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:11.953420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:11.953474 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:11.953642 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:11.953643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:11.953646 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:11.953935 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.953937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.954544 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.954545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:11.955097 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.955098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.955192 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.955196 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:11.955225 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:08:11.955259 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260201 03:08:11.955677 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.953346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:43579 (local address 127.28.181.1:37471)
0201 03:08:11.953454 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.953456 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.953547 (+ 91us) server_negotiation.cc:408] Connection header received
0201 03:08:11.953623 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.953625 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.953635 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.953726 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:11.954060 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.954497 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.955156 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.955535 (+ 379us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.955544 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.955584 (+ 40us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.955616 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260201 03:08:11.955780 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.953388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:44469 (local address 127.28.181.3:40835)
0201 03:08:11.953449 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.953451 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.953477 (+ 26us) server_negotiation.cc:408] Connection header received
0201 03:08:11.953541 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.953542 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.953561 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.953628 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:11.954023 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.954455 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.955175 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.955638 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.955647 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.955671 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.955706 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:08:11.955906 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "135dac9cd2924c4298c4ee2c5b0feba0" candidate_uuid: "67d15bb1aceb45f7b36518caf476a68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9" is_pre_election: true
I20260201 03:08:11.955992 31302 raft_consensus.cc:2468] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d15bb1aceb45f7b36518caf476a68e in term 0.
I20260201 03:08:11.955941 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "135dac9cd2924c4298c4ee2c5b0feba0" candidate_uuid: "67d15bb1aceb45f7b36518caf476a68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" is_pre_election: true
I20260201 03:08:11.956029 31586 raft_consensus.cc:2468] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d15bb1aceb45f7b36518caf476a68e in term 0.
I20260201 03:08:11.956183 31376 leader_election.cc:304] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:11.956315 31747 raft_consensus.cc:2804] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:11.956367 31747 raft_consensus.cc:493] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:11.956398 31747 raft_consensus.cc:3060] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.956960 31747 raft_consensus.cc:515] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.957098 31747 leader_election.cc:290] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [CANDIDATE]: Term 1 election: Requested vote from peers a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471), 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835)
I20260201 03:08:11.957247 31302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "135dac9cd2924c4298c4ee2c5b0feba0" candidate_uuid: "67d15bb1aceb45f7b36518caf476a68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25056fed19b44f2902a3462ba9d17d9"
I20260201 03:08:11.957310 31302 raft_consensus.cc:3060] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.957844 31586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "135dac9cd2924c4298c4ee2c5b0feba0" candidate_uuid: "67d15bb1aceb45f7b36518caf476a68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb"
I20260201 03:08:11.957897 31586 raft_consensus.cc:3060] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:11.957922 31302 raft_consensus.cc:2468] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d15bb1aceb45f7b36518caf476a68e in term 1.
I20260201 03:08:11.958063 31376 leader_election.cc:304] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d15bb1aceb45f7b36518caf476a68e, a25056fed19b44f2902a3462ba9d17d9; no voters:
I20260201 03:08:11.958168 31747 raft_consensus.cc:2804] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:11.958319 31747 raft_consensus.cc:697] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 1 LEADER]: Becoming Leader. State: Replica: 67d15bb1aceb45f7b36518caf476a68e, State: Running, Role: LEADER
I20260201 03:08:11.958381 31747 consensus_queue.cc:237] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } }
I20260201 03:08:11.958705 31586 raft_consensus.cc:2468] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d15bb1aceb45f7b36518caf476a68e in term 1.
I20260201 03:08:11.959097 31141 catalog_manager.cc:5654] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2). New cstate: current_term: 1 leader_uuid: "67d15bb1aceb45f7b36518caf476a68e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d15bb1aceb45f7b36518caf476a68e" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:11.978360 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.976464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.28.181.2:33491)
0201 03:08:11.976596 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.976597 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.976605 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:11.976630 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.976631 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.976644 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.976712 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:11.976927 (+ 215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.977325 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.977872 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.978030 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.978058 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.978141 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.978225 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.978256 (+ 31us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.978278 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260201 03:08:11.981732 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.980280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:59021 (local address 127.28.181.1:37471)
0201 03:08:11.980328 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.980330 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.980344 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:08:11.980377 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.980377 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.980385 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.980420 (+ 35us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:11.980632 (+ 212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.980999 (+ 367us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.981462 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.981549 (+ 87us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.981561 (+ 12us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.981582 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.981617 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.981681 (+ 64us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.981696 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260201 03:08:11.981741 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.980125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.28.181.2:59021)
0201 03:08:11.980290 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:11.980298 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:11.980338 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:11.980436 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:11.980437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:11.980439 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:11.980581 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.980582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.981056 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.981056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:11.981398 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.981399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.981457 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.981467 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.981634 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.981635 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:11.981668 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:08:11.981694 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260201 03:08:11.983362 31586 raft_consensus.cc:1275] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Refusing update from remote peer 67d15bb1aceb45f7b36518caf476a68e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:11.983361 31302 raft_consensus.cc:1275] T 135dac9cd2924c4298c4ee2c5b0feba0 P a25056fed19b44f2902a3462ba9d17d9 [term 1 FOLLOWER]: Refusing update from remote peer 67d15bb1aceb45f7b36518caf476a68e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:11.983614 31762 consensus_queue.cc:1048] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25056fed19b44f2902a3462ba9d17d9" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:11.983752 31747 consensus_queue.cc:1048] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8700a7d3fbf741f5a68b01ca5ed043fb" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:11.989605 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.987800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33491 (local address 127.28.181.1:35945)
0201 03:08:11.987917 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:11.987925 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:11.988002 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:11.988152 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:11.988152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:11.988158 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:08:11.988318 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.988319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.988925 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.988925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:11.989243 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:11.989244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.989295 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.989298 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.989478 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.989479 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:11.989508 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:08:11.989550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":80,"threads_started":1}
I20260201 03:08:11.989617 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:11.987841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:35945 (local address 127.28.181.2:33491)
0201 03:08:11.987872 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:11.987874 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:11.988011 (+ 137us) server_negotiation.cc:408] Connection header received
0201 03:08:11.988059 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:11.988059 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:11.988068 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:11.988113 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:11.988391 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.988865 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:11.989301 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:11.989402 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:11.989414 (+ 12us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:11.989429 (+ 15us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:11.989465 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:11.989565 (+ 100us) server_negotiation.cc:300] Negotiation successful
0201 03:08:11.989580 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260201 03:08:11.989917 31714 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260201 03:08:13.508268 31142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:13.508371 31142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:13.509095 31142 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2a1c3b38f40548d29e3e1af4ae3ec231: Sending DeleteTablet for 3 replicas of tablet 135dac9cd2924c4298c4ee2c5b0feba0
I20260201 03:08:13.509302 31282 tablet_service.cc:1552] Processing DeleteTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:13 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260201 03:08:13.509387 31424 tablet_service.cc:1552] Processing DeleteTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:13 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260201 03:08:13.509387 31551 tablet_service.cc:1552] Processing DeleteTablet for tablet 135dac9cd2924c4298c4ee2c5b0feba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:13 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:52776
I20260201 03:08:13.509555 31797 tablet_replica.cc:333] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: stopping tablet replica
I20260201 03:08:13.509562 31796 tablet_replica.cc:333] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: stopping tablet replica
I20260201 03:08:13.509608 31796 raft_consensus.cc:2243] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:13.509635 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31217
I20260201 03:08:13.510473 31796 raft_consensus.cc:2272] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:13.510718 31797 raft_consensus.cc:2243] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:13.510746 31796 ts_tablet_manager.cc:1916] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:13.510775 31797 raft_consensus.cc:2272] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:13.511001 31797 ts_tablet_manager.cc:1916] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:13.512364 31797 ts_tablet_manager.cc:1929] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:08:13.512408 31797 log.cc:1199] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/135dac9cd2924c4298c4ee2c5b0feba0
I20260201 03:08:13.512526 31796 ts_tablet_manager.cc:1929] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:08:13.512579 31796 log.cc:1199] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/135dac9cd2924c4298c4ee2c5b0feba0
I20260201 03:08:13.512631 31797 ts_tablet_manager.cc:1950] T 135dac9cd2924c4298c4ee2c5b0feba0 P 8700a7d3fbf741f5a68b01ca5ed043fb: Deleting consensus metadata
I20260201 03:08:13.512833 31796 ts_tablet_manager.cc:1950] T 135dac9cd2924c4298c4ee2c5b0feba0 P 67d15bb1aceb45f7b36518caf476a68e: Deleting consensus metadata
I20260201 03:08:13.512923 31129 catalog_manager.cc:4985] TS 8700a7d3fbf741f5a68b01ca5ed043fb (127.28.181.3:40835): tablet 135dac9cd2924c4298c4ee2c5b0feba0 (table test-table [id=b59fa6612fb044b49636fd6f672906f1]) successfully deleted
I20260201 03:08:13.513144 31127 catalog_manager.cc:4985] TS 67d15bb1aceb45f7b36518caf476a68e (127.28.181.2:33491): tablet 135dac9cd2924c4298c4ee2c5b0feba0 (table test-table [id=b59fa6612fb044b49636fd6f672906f1]) successfully deleted
W20260201 03:08:13.516009 31128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.181.1:37471 (error 108)
I20260201 03:08:13.516599 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:13.516359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:37471 (local address 127.0.0.1:59296)
0201 03:08:13.516494 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:13.516531 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:37471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260201 03:08:13.516695 31128 catalog_manager.cc:4712] TS a25056fed19b44f2902a3462ba9d17d9 (127.28.181.1:37471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 135dac9cd2924c4298c4ee2c5b0feba0: Network error: Client connection negotiation failed: client connection to 127.28.181.1:37471: connect: Connection refused (error 111)
I20260201 03:08:13.516975 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31359
I20260201 03:08:13.522747 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31501
I20260201 03:08:13.527942 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31111
2026-02-01T03:08:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6659 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:13 dist-test-slave-ck1n krb5kdc[31802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:13 dist-test-slave-ck1n krb5kdc[31802](info): set up 2 sockets
Feb 01 03:08:13 dist-test-slave-ck1n krb5kdc[31802](info): commencing operation
krb5kdc: starting...
W20260201 03:08:15.259630 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.712s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:15.365666 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:33849
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:15.432015 31818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:15.432148 31818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:15.432166 31818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:15.433353 31818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:15.433390 31818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:15.433403 31818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:15.433413 31818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:15.433459 31818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:15.434945 31818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:15.435140 31818 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:15.435340 31818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:08:15.437544 31818 server_base.cc:1060] running on GCE node
W20260201 03:08:15.437526 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.437505 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.437510 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:15.437922 31818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:15.438086 31818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:15.439205 31818 hybrid_clock.cc:648] HybridClock initialized: now 1769915295439198 us; error 23 us; skew 500 ppm
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:15.440992 31818 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:15.441444 31818 webserver.cc:492] Webserver started at http://127.28.181.62:34249/ using document root <none> and password file <none>
I20260201 03:08:15.441608 31818 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:15.441655 31818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:15.441759 31818 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:15.442531 31818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "2ad475b0de3d642925c4c2f9647d99be"
server_key_iv: "2b43426ac84da84b6bda8d62110d87b4"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.442818 31818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "2ad475b0de3d642925c4c2f9647d99be"
server_key_iv: "2b43426ac84da84b6bda8d62110d87b4"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.443787 31818 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:15.444378 31833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.444527 31818 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:15.444604 31818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "2ad475b0de3d642925c4c2f9647d99be"
server_key_iv: "2b43426ac84da84b6bda8d62110d87b4"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.444671 31818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:15.459972 31818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:15.464896 31818 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:15.465024 31818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:15.468747 31818 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:33849
I20260201 03:08:15.468775 31885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:33849 every 8 connection(s)
I20260201 03:08:15.469012 31818 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:15.469447 31886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:15.471467 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b: Bootstrap starting.
I20260201 03:08:15.471956 31886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:15.472170 31886 log.cc:826] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:15.472739 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b: No bootstrap required, opened a new log
I20260201 03:08:15.473690 31886 raft_consensus.cc:359] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } }
I20260201 03:08:15.473830 31886 raft_consensus.cc:385] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:15.473887 31886 raft_consensus.cc:740] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08ab2f15b2f4e6d9510d4b4822ce75b, State: Initialized, Role: FOLLOWER
I20260201 03:08:15.474009 31886 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } }
I20260201 03:08:15.474077 31886 raft_consensus.cc:399] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:15.474112 31886 raft_consensus.cc:493] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:15.474157 31886 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:15.474618 31886 raft_consensus.cc:515] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } }
I20260201 03:08:15.474722 31886 leader_election.cc:304] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08ab2f15b2f4e6d9510d4b4822ce75b; no voters:
I20260201 03:08:15.474862 31886 leader_election.cc:290] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:15.474911 31889 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:15.475065 31886 sys_catalog.cc:565] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:15.475075 31889 raft_consensus.cc:697] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [term 1 LEADER]: Becoming Leader. State: Replica: f08ab2f15b2f4e6d9510d4b4822ce75b, State: Running, Role: LEADER
I20260201 03:08:15.475209 31889 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } }
I20260201 03:08:15.475543 31891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ab2f15b2f4e6d9510d4b4822ce75b. Latest consensus state: current_term: 1 leader_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } } }
I20260201 03:08:15.475627 31891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:15.475809 31898 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:15.475977 31890 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab2f15b2f4e6d9510d4b4822ce75b" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 33849 } } }
I20260201 03:08:15.476048 31890 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:15.476409 31898 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:15.477550 31898 catalog_manager.cc:1357] Generated new cluster ID: 1175989c923e469e90b413b62335b845
I20260201 03:08:15.477586 31898 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:15.478899 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31818
I20260201 03:08:15.478987 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:15.479105 29396 external_mini_cluster.cc:1468] Setting key 00fe5f9af4174e030feee8d34e57b394
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:15.487854 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.479474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.28.181.62:33849)
0201 03:08:15.479621 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:15.479630 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:15.479648 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:08:15.479729 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:15.479733 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:15.479768 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:15.479872 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:15.480659 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.481607 (+ 948us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.482310 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.482502 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.484575 (+ 2073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:15.484581 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:15.484581 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:15.484599 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:15.486696 (+ 2097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.487011 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.487014 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.487014 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.487077 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.487202 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.487203 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.487203 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.487388 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:15.487517 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:15.487595 (+ 78us) server_negotiation.cc:300] Negotiation successful
0201 03:08:15.487698 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260201 03:08:15.499594 31898 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:15.500195 31898 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:15.507740 31898 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f08ab2f15b2f4e6d9510d4b4822ce75b: Generated new TSK 0
I20260201 03:08:15.507881 31898 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:15.549228 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:33849
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:15.615306 31914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:15.615427 31914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:15.615445 31914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:15.616643 31914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:15.616683 31914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:15.616707 31914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:15.618106 31914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:15.618274 31914 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:15.618433 31914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:15.620364 31920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.620393 31919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.620396 31922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:15.620538 31914 server_base.cc:1060] running on GCE node
I20260201 03:08:15.620723 31914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:15.620901 31914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:15.622035 31914 hybrid_clock.cc:648] HybridClock initialized: now 1769915295622025 us; error 26 us; skew 500 ppm
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:15.623967 31914 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:15.624409 31914 webserver.cc:492] Webserver started at http://127.28.181.1:34645/ using document root <none> and password file <none>
I20260201 03:08:15.624534 31914 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:15.624560 31914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:15.624621 31914 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:15.625312 31914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "5f9eb739c1174642841c5477695de37a"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "0667e2b315145ef5f68b3a592ef9a160"
server_key_iv: "948592490772ae087737498f7c23666f"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.625527 31914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "5f9eb739c1174642841c5477695de37a"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "0667e2b315145ef5f68b3a592ef9a160"
server_key_iv: "948592490772ae087737498f7c23666f"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.626531 31914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:15.627066 31929 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.627223 31914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:15.627282 31914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "5f9eb739c1174642841c5477695de37a"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "0667e2b315145ef5f68b3a592ef9a160"
server_key_iv: "948592490772ae087737498f7c23666f"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.627321 31914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:15.648871 31914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:15.650573 31914 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:15.650679 31914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:15.650828 31914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:15.651041 31914 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:15.651065 31914 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.651082 31914 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:15.651094 31914 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.656075 31914 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:44835
I20260201 03:08:15.656150 32042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:44835 every 8 connection(s)
I20260201 03:08:15.656366 31914 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:15.662371 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 31914
I20260201 03:08:15.662480 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:15.662698 29396 external_mini_cluster.cc:1468] Setting key 2c4dc8993f3e74dfdca1107304d38b4a
I20260201 03:08:15.662887 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.656962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:58669 (local address 127.28.181.62:33849)
0201 03:08:15.657001 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:15.657003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:15.657115 (+ 112us) server_negotiation.cc:408] Connection header received
0201 03:08:15.657513 (+ 398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:15.657514 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:15.657524 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:15.657573 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:15.658205 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.658637 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.659104 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.659205 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.660774 (+ 1569us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:15.660777 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:15.660777 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:15.660783 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:15.661755 (+ 972us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.662081 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.662082 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.662082 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.662100 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.662261 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.662262 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.662262 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.662360 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:15.662424 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:15.662722 (+ 298us) server_negotiation.cc:300] Negotiation successful
0201 03:08:15.662807 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260201 03:08:15.663000 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.656908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:33849 (local address 127.28.181.1:58669)
0201 03:08:15.657033 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:15.657060 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:15.657436 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:15.657646 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:15.657649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:15.657910 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:15.658135 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.658137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.658706 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.658706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:15.659044 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.659045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.659313 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.659631 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:15.659644 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:15.660705 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:15.661859 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.661864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.661865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.662017 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.662147 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.662147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.662147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.662190 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.662416 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:15.662419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:15.662584 (+ 165us) client_negotiation.cc:769] Sending connection context
0201 03:08:15.662687 (+ 103us) client_negotiation.cc:240] Negotiation successful
0201 03:08:15.662818 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260201 03:08:15.663462 32043 heartbeater.cc:344] Connected to a master server at 127.28.181.62:33849
I20260201 03:08:15.663544 32043 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:15.663727 32043 heartbeater.cc:507] Master 127.28.181.62:33849 requested a full tablet report, sending...
I20260201 03:08:15.664156 31850 ts_manager.cc:194] Registered new tserver with Master: 5f9eb739c1174642841c5477695de37a (127.28.181.1:44835)
I20260201 03:08:15.664973 31850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:58669
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:15.699625 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:33849
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:15.765092 32050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:15.765220 32050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:15.765240 32050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:15.766597 32050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:15.766657 32050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:15.766681 32050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:15.768074 32050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:15.768231 32050 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:15.768406 32050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:15.770362 32056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.770360 32058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.770468 32055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:15.770661 32050 server_base.cc:1060] running on GCE node
I20260201 03:08:15.770874 32050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:15.771037 32050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:15.772157 32050 hybrid_clock.cc:648] HybridClock initialized: now 1769915295772139 us; error 35 us; skew 500 ppm
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:15.774564 32050 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:15.775010 32050 webserver.cc:492] Webserver started at http://127.28.181.2:46519/ using document root <none> and password file <none>
I20260201 03:08:15.775151 32050 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:15.775177 32050 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:15.775236 32050 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:15.775985 32050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "770a73d7c9bc44bca3a001c707f675bc"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "192eb4528220b240641f48aa24f8b771"
server_key_iv: "9bd0114bc8cae51723376bc915db33a7"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.776227 32050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "770a73d7c9bc44bca3a001c707f675bc"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "192eb4528220b240641f48aa24f8b771"
server_key_iv: "9bd0114bc8cae51723376bc915db33a7"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.777143 32050 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:15.777640 32065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.777845 32050 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.777911 32050 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "770a73d7c9bc44bca3a001c707f675bc"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "192eb4528220b240641f48aa24f8b771"
server_key_iv: "9bd0114bc8cae51723376bc915db33a7"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.777978 32050 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:15.788813 32050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:15.790604 32050 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:15.790736 32050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:15.790920 32050 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:15.791167 32050 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:15.791213 32050 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.791247 32050 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:15.791271 32050 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.796177 32050 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:40437
I20260201 03:08:15.796206 32178 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:40437 every 8 connection(s)
I20260201 03:08:15.796440 32050 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:15.802734 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32050
I20260201 03:08:15.802937 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:15.802951 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.796958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:33849 (local address 127.28.181.2:60055)
0201 03:08:15.797072 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:15.797098 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:15.797443 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:15.797594 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:15.797596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:15.797877 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:15.798112 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.798115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.798654 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.798655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:15.798983 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.798984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.799261 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.799590 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:15.799603 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:15.800666 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:15.801912 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.801916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.801919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.802096 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.802232 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.802233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.802233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.802280 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.802448 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:15.802451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:15.802541 (+ 90us) client_negotiation.cc:769] Sending connection context
0201 03:08:15.802668 (+ 127us) client_negotiation.cc:240] Negotiation successful
0201 03:08:15.802794 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260201 03:08:15.802923 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.796993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:60055 (local address 127.28.181.62:33849)
0201 03:08:15.797041 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:15.797043 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:15.797160 (+ 117us) server_negotiation.cc:408] Connection header received
0201 03:08:15.797504 (+ 344us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:15.797505 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:15.797512 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:15.797559 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:15.798181 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.798588 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.799045 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.799135 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.800740 (+ 1605us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:15.800743 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:15.800743 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:15.800749 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:15.801844 (+ 1095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.802159 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.802160 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.802160 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.802179 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.802328 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.802328 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.802329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.802400 (+ 71us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:15.802446 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:15.802706 (+ 260us) server_negotiation.cc:300] Negotiation successful
0201 03:08:15.802805 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:08:15.803109 29396 external_mini_cluster.cc:1468] Setting key 33049e78a80a986a4e3562800ed29d5b
I20260201 03:08:15.803360 32179 heartbeater.cc:344] Connected to a master server at 127.28.181.62:33849
I20260201 03:08:15.803437 32179 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:15.803614 32179 heartbeater.cc:507] Master 127.28.181.62:33849 requested a full tablet report, sending...
I20260201 03:08:15.803956 31850 ts_manager.cc:194] Registered new tserver with Master: 770a73d7c9bc44bca3a001c707f675bc (127.28.181.2:40437)
I20260201 03:08:15.804311 31850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:60055
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:15.838426 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:33849
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:15.904291 32186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:15.904449 32186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:15.904481 32186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:15.905722 32186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:15.905782 32186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:15.905849 32186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:15.907209 32186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:15.907423 32186 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:15.907581 32186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:15.909467 32191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.909466 32192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:15.909497 32194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:15.909633 32186 server_base.cc:1060] running on GCE node
I20260201 03:08:15.909852 32186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:15.910058 32186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:15.911191 32186 hybrid_clock.cc:648] HybridClock initialized: now 1769915295911183 us; error 30 us; skew 500 ppm
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:15.913321 32186 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:15.913763 32186 webserver.cc:492] Webserver started at http://127.28.181.3:34493/ using document root <none> and password file <none>
I20260201 03:08:15.914007 32186 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:15.914058 32186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:15.914161 32186 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:15.914939 32186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "f738850b66ec49a9a0478250696f4500"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "ef3a06ba7bfef9937e23e48df45e65f4"
server_key_iv: "64b9246651b80a46806e24aac166d9ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.915206 32186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "f738850b66ec49a9a0478250696f4500"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "ef3a06ba7bfef9937e23e48df45e65f4"
server_key_iv: "64b9246651b80a46806e24aac166d9ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.916168 32186 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:15.916707 32201 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.916846 32186 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:15.916924 32186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "f738850b66ec49a9a0478250696f4500"
format_stamp: "Formatted at 2026-02-01 03:08:15 on dist-test-slave-ck1n"
server_key: "ef3a06ba7bfef9937e23e48df45e65f4"
server_key_iv: "64b9246651b80a46806e24aac166d9ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:15.916989 32186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:15.944715 32186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:15.946367 32186 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:15.946491 32186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:15.946668 32186 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:15.946925 32186 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:15.946971 32186 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.947022 32186 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:15.947050 32186 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:15.951712 32186 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:41185
I20260201 03:08:15.951771 32314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:41185 every 8 connection(s)
I20260201 03:08:15.951998 32186 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:08:15 dist-test-slave-ck1n krb5kdc[31802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:15.958333 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.952568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:46251 (local address 127.28.181.62:33849)
0201 03:08:15.952638 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:15.952640 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:15.952707 (+ 67us) server_negotiation.cc:408] Connection header received
0201 03:08:15.953056 (+ 349us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:15.953058 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:15.953066 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:15.953106 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:15.953655 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.954142 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.954610 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.954708 (+ 98us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.956328 (+ 1620us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:15.956331 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:15.956332 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:15.956337 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:15.957493 (+ 1156us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.957759 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.957759 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.957760 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.957790 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:15.957941 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:15.957941 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:15.957942 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:15.958006 (+ 64us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:15.958045 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:15.958221 (+ 176us) server_negotiation.cc:300] Negotiation successful
0201 03:08:15.958278 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:08:15.958417 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:15.952516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:33849 (local address 127.28.181.3:46251)
0201 03:08:15.952614 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:15.952642 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:15.952997 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:15.953145 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:15.953148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:15.953367 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:15.953587 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.953589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.954206 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:15.954206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:15.954532 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:15.954533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:15.954838 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:15.955197 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:15.955210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:15.956255 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:15.957559 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.957562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.957564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.957704 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.957855 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:15.957855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:15.957856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:15.957899 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:15.958048 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:15.958050 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:15.958130 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:08:15.958211 (+ 81us) client_negotiation.cc:240] Negotiation successful
0201 03:08:15.958287 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":45,"threads_started":1}
I20260201 03:08:15.958829 32315 heartbeater.cc:344] Connected to a master server at 127.28.181.62:33849
I20260201 03:08:15.958896 32315 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:15.959015 32315 heartbeater.cc:507] Master 127.28.181.62:33849 requested a full tablet report, sending...
I20260201 03:08:15.959286 31850 ts_manager.cc:194] Registered new tserver with Master: f738850b66ec49a9a0478250696f4500 (127.28.181.3:41185)
I20260201 03:08:15.959573 31850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:46251
I20260201 03:08:15.961685 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32186
I20260201 03:08:15.961800 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:15.961953 29396 external_mini_cluster.cc:1468] Setting key c5102c9051d4d3b95409cea7de744fde
I20260201 03:08:15.962687 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:08:15.971378 31909 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:15.970634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.28.181.62:33849)
0201 03:08:15.970700 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:15.970702 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:15.970792 (+ 90us) server_negotiation.cc:408] Connection header received
0201 03:08:15.970862 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:15.970862 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:15.970872 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:15.970941 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:15.971343 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34502: BlockingRecv error: recv got EOF from 127.0.0.1:34502 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260201 03:08:15.972469 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31914
I20260201 03:08:15.976456 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32050
I20260201 03:08:15.980434 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32186
I20260201 03:08:15.983843 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 31818
2026-02-01T03:08:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2454 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:16 dist-test-slave-ck1n krb5kdc[32327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:16 dist-test-slave-ck1n krb5kdc[32327](info): set up 2 sockets
Feb 01 03:08:16 dist-test-slave-ck1n krb5kdc[32327](info): commencing operation
krb5kdc: starting...
W20260201 03:08:17.695508 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.693s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:17 dist-test-slave-ck1n krb5kdc[32327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:17.802001 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:39395
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:17.867777 32343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:17.867913 32343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:17.867933 32343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:17.869041 32343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:17.869073 32343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:17.869086 32343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:17.869096 32343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:17.869107 32343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:17.870368 32343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:39395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:17.870507 32343 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:17.870607 32343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:17.872418 32348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:17.872418 32351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:17.872457 32349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:17.872548 32343 server_base.cc:1060] running on GCE node
I20260201 03:08:17.872834 32343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:17.873006 32343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:17.874155 32343 hybrid_clock.cc:648] HybridClock initialized: now 1769915297874144 us; error 22 us; skew 500 ppm
Feb 01 03:08:17 dist-test-slave-ck1n krb5kdc[32327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:17.876112 32343 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:17.876539 32343 webserver.cc:492] Webserver started at http://127.28.181.62:36653/ using document root <none> and password file <none>
I20260201 03:08:17.876706 32343 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:17.876737 32343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:17.876842 32343 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:17.877573 32343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "f9d552b2a3964261bc7b109c84bd05a3"
format_stamp: "Formatted at 2026-02-01 03:08:17 on dist-test-slave-ck1n"
server_key: "41e84cf0a8be9fad14513f9363ac41ba"
server_key_iv: "fb56a10b990f0eb22d62cc802cf5c4ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:17.877897 32343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "f9d552b2a3964261bc7b109c84bd05a3"
format_stamp: "Formatted at 2026-02-01 03:08:17 on dist-test-slave-ck1n"
server_key: "41e84cf0a8be9fad14513f9363ac41ba"
server_key_iv: "fb56a10b990f0eb22d62cc802cf5c4ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:17.878870 32343 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:17.879417 32358 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:17.879572 32343 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:17.879627 32343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "f9d552b2a3964261bc7b109c84bd05a3"
format_stamp: "Formatted at 2026-02-01 03:08:17 on dist-test-slave-ck1n"
server_key: "41e84cf0a8be9fad14513f9363ac41ba"
server_key_iv: "fb56a10b990f0eb22d62cc802cf5c4ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:17.879698 32343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:17.902029 32343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:17.903617 32343 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:17.903740 32343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:17.907123 32343 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:39395
I20260201 03:08:17.907186 32410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:39395 every 8 connection(s)
I20260201 03:08:17.907384 32343 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:17.907895 32411 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:17.909765 32411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: Bootstrap starting.
I20260201 03:08:17.910250 32411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:17.910468 32411 log.cc:826] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:17.911034 32411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: No bootstrap required, opened a new log
I20260201 03:08:17.911985 32411 raft_consensus.cc:359] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } }
I20260201 03:08:17.912076 32411 raft_consensus.cc:385] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:17.912093 32411 raft_consensus.cc:740] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d552b2a3964261bc7b109c84bd05a3, State: Initialized, Role: FOLLOWER
I20260201 03:08:17.912210 32411 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } }
I20260201 03:08:17.912263 32411 raft_consensus.cc:399] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:17.912281 32411 raft_consensus.cc:493] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:17.912333 32411 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:17.912746 32411 raft_consensus.cc:515] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } }
I20260201 03:08:17.912843 32411 leader_election.cc:304] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9d552b2a3964261bc7b109c84bd05a3; no voters:
I20260201 03:08:17.912976 32411 leader_election.cc:290] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:17.913013 32414 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:17.913137 32414 raft_consensus.cc:697] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [term 1 LEADER]: Becoming Leader. State: Replica: f9d552b2a3964261bc7b109c84bd05a3, State: Running, Role: LEADER
I20260201 03:08:17.913158 32411 sys_catalog.cc:565] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:17.913239 32414 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } }
I20260201 03:08:17.913543 32416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9d552b2a3964261bc7b109c84bd05a3. Latest consensus state: current_term: 1 leader_uuid: "f9d552b2a3964261bc7b109c84bd05a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } } }
I20260201 03:08:17.913529 32415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9d552b2a3964261bc7b109c84bd05a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d552b2a3964261bc7b109c84bd05a3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39395 } } }
I20260201 03:08:17.913692 32415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:17.914012 32416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3 [sys.catalog]: This master's current role is: LEADER
W20260201 03:08:17.914868 32430 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:17.914922 32430 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:17.914983 32422 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:17.915115 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32343
I20260201 03:08:17.915143 32422 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:17.915197 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:17.915342 29396 external_mini_cluster.cc:1468] Setting key 6bc266da8294b5873e7b15b949866b90
I20260201 03:08:17.916319 32422 catalog_manager.cc:1357] Generated new cluster ID: 114cb470fbfe44b7972be40606d91534
I20260201 03:08:17.916365 32422 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:08:17 dist-test-slave-ck1n krb5kdc[32327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:17.925747 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:17.915858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.28.181.62:39395)
0201 03:08:17.915987 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:17.915998 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:17.916015 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:08:17.916098 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:17.916103 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:17.916142 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:17.916247 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:17.918522 (+ 2275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:17.919530 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:17.920193 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:17.920448 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:17.922411 (+ 1963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:17.922415 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:17.922415 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:17.922432 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:17.924525 (+ 2093us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:17.924873 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:17.924876 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:17.924876 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:17.924923 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:17.925084 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:17.925084 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:17.925085 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:17.925273 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:17.925376 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:17.925519 (+ 143us) server_negotiation.cc:300] Negotiation successful
0201 03:08:17.925619 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260201 03:08:17.930371 32422 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:17.930791 32422 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:17.934310 32422 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f9d552b2a3964261bc7b109c84bd05a3: Generated new TSK 0
I20260201 03:08:17.934455 32422 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:17.972193 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:39395
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:18.037653 32439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:18.037832 32439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:18.037868 32439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:18.039038 32439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:18.039096 32439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:18.039134 32439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:18.040431 32439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:18.040647 32439 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:18.040800 32439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:18.042712 32444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:18.042718 32447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:18.042742 32445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:18.043036 32439 server_base.cc:1060] running on GCE node
I20260201 03:08:18.043179 32439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:18.043359 32439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:18.044497 32439 hybrid_clock.cc:648] HybridClock initialized: now 1769915298044483 us; error 39 us; skew 500 ppm
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:18.046379 32439 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:18.046751 32439 webserver.cc:492] Webserver started at http://127.28.181.1:36237/ using document root <none> and password file <none>
I20260201 03:08:18.046871 32439 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:18.046896 32439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:18.046967 32439 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:18.047662 32439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "581766ef054c46e6a920275b57788f05"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "8647e85d44d1b970af686dcb99c17749"
server_key_iv: "39c4ee24de0e5186b5f4d60e92b896e3"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.047878 32439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "581766ef054c46e6a920275b57788f05"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "8647e85d44d1b970af686dcb99c17749"
server_key_iv: "39c4ee24de0e5186b5f4d60e92b896e3"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.048813 32439 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:18.049346 32454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.049507 32439 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.049558 32439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "581766ef054c46e6a920275b57788f05"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "8647e85d44d1b970af686dcb99c17749"
server_key_iv: "39c4ee24de0e5186b5f4d60e92b896e3"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.049592 32439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:18.059854 32439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:18.061921 32439 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:18.062078 32439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:18.062269 32439 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:18.062601 32439 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:18.062647 32439 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.062676 32439 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:18.062703 32439 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.070323 32439 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:45689
I20260201 03:08:18.070355 32567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:45689 every 8 connection(s)
I20260201 03:08:18.070712 32439 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:18.075389 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32439
I20260201 03:08:18.075475 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:18.075626 29396 external_mini_cluster.cc:1468] Setting key ac6dc2776efb935a854247e1b3eb5d63
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:18.079257 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.071539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49573 (local address 127.28.181.62:39395)
0201 03:08:18.071587 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.071589 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.071597 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:18.071985 (+ 388us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.071987 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.071999 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.072048 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.072921 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.073476 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.074199 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.074389 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.076522 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:18.076525 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:18.076525 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:18.076531 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:18.077950 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.078354 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.078356 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.078356 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.078383 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.078606 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.078607 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.078607 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.078710 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:18.078762 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:18.079038 (+ 276us) server_negotiation.cc:300] Negotiation successful
0201 03:08:18.079134 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260201 03:08:18.079845 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.071307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:39395 (local address 127.28.181.1:49573)
0201 03:08:18.071477 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:18.071511 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:18.071878 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:18.072115 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:18.072118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:18.072468 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:18.072830 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.072834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.073551 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.073552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:18.074103 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.074105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.074223 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.075037 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:18.075050 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:18.076444 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:18.078028 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.078034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.078036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.078293 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.078460 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.078461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.078462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.078546 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.078761 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:18.078766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:18.078891 (+ 125us) client_negotiation.cc:769] Sending connection context
0201 03:08:18.079513 (+ 622us) client_negotiation.cc:240] Negotiation successful
0201 03:08:18.079646 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260201 03:08:18.080364 32568 heartbeater.cc:344] Connected to a master server at 127.28.181.62:39395
I20260201 03:08:18.080466 32568 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:18.080642 32568 heartbeater.cc:507] Master 127.28.181.62:39395 requested a full tablet report, sending...
I20260201 03:08:18.081054 32375 ts_manager.cc:194] Registered new tserver with Master: 581766ef054c46e6a920275b57788f05 (127.28.181.1:45689)
I20260201 03:08:18.081919 32375 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:49573
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:18.114006 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:39395
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:18.178828 32575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:18.178948 32575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:18.178966 32575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:18.180063 32575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:18.180096 32575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:18.180117 32575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:18.181331 32575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:18.181454 32575 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:18.181557 32575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:18.183498 32583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:18.183477 32580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:18.183475 32581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:18.183698 32575 server_base.cc:1060] running on GCE node
I20260201 03:08:18.183808 32575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:18.183974 32575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:18.185088 32575 hybrid_clock.cc:648] HybridClock initialized: now 1769915298185073 us; error 29 us; skew 500 ppm
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:18.186926 32575 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:18.187346 32575 webserver.cc:492] Webserver started at http://127.28.181.2:43543/ using document root <none> and password file <none>
I20260201 03:08:18.187469 32575 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:18.187495 32575 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:18.187554 32575 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:18.188230 32575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "e9828d7290e145399f678dadfcc02a24"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "fc80cdabd6b9afa4df9b18f1d4214f26"
server_key_iv: "424a0fe9593c4f0309ef86457de41202"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.188437 32575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "e9828d7290e145399f678dadfcc02a24"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "fc80cdabd6b9afa4df9b18f1d4214f26"
server_key_iv: "424a0fe9593c4f0309ef86457de41202"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.189333 32575 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:18.189877 32590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.190025 32575 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.190104 32575 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "e9828d7290e145399f678dadfcc02a24"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "fc80cdabd6b9afa4df9b18f1d4214f26"
server_key_iv: "424a0fe9593c4f0309ef86457de41202"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.190165 32575 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:18.198658 32575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:18.200187 32575 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:18.200304 32575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:18.200479 32575 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:18.200714 32575 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:18.200759 32575 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.200793 32575 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:18.200814 32575 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.206071 32575 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:39057
I20260201 03:08:18.206099 32703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:39057 every 8 connection(s)
I20260201 03:08:18.206317 32575 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:18.206840 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32575
I20260201 03:08:18.206938 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:18.207119 29396 external_mini_cluster.cc:1468] Setting key d6aae781fc93858ef5b132dbfe0b650c
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
I20260201 03:08:18.213987 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.206854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:57649 (local address 127.28.181.62:39395)
0201 03:08:18.206910 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.206912 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.207090 (+ 178us) server_negotiation.cc:408] Connection header received
0201 03:08:18.207557 (+ 467us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.207558 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.207567 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.207634 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.208418 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.208955 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.209510 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.209620 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.211706 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:18.211709 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:18.211709 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:18.211720 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:18.212868 (+ 1148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.213238 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.213239 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.213239 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.213266 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.213476 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.213477 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.213478 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.213580 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:18.213652 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:18.213873 (+ 221us) server_negotiation.cc:300] Negotiation successful
0201 03:08:18.213934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:08:18.214082 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.206827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:39395 (local address 127.28.181.2:57649)
0201 03:08:18.206974 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:18.207021 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:18.207480 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:18.207683 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:18.207685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:18.208023 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:18.208341 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.208344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.209035 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.209036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:18.209433 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.209435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.209544 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.210359 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:18.210379 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:18.211622 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:18.212977 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.212982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.212984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.213162 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.213320 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.213321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.213322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.213391 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.213645 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:18.213649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:18.213731 (+ 82us) client_negotiation.cc:769] Sending connection context
0201 03:08:18.213848 (+ 117us) client_negotiation.cc:240] Negotiation successful
0201 03:08:18.213929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260201 03:08:18.214522 32704 heartbeater.cc:344] Connected to a master server at 127.28.181.62:39395
I20260201 03:08:18.214624 32704 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:18.214797 32704 heartbeater.cc:507] Master 127.28.181.62:39395 requested a full tablet report, sending...
I20260201 03:08:18.215097 32375 ts_manager.cc:194] Registered new tserver with Master: e9828d7290e145399f678dadfcc02a24 (127.28.181.2:39057)
I20260201 03:08:18.215378 32375 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:57649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:18.242264 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:39395
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:18.306670 32711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:18.306797 32711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:18.306816 32711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:18.308101 32711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:18.308138 32711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:18.308161 32711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:18.309484 32711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:18.309656 32711 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:18.309859 32711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:18.311913 32717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:18.311960 32711 server_base.cc:1060] running on GCE node
W20260201 03:08:18.311904 32719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:18.311901 32716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:18.312286 32711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:18.312469 32711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:18.313607 32711 hybrid_clock.cc:648] HybridClock initialized: now 1769915298313591 us; error 32 us; skew 500 ppm
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:18.315989 32711 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:18.316488 32711 webserver.cc:492] Webserver started at http://127.28.181.3:34801/ using document root <none> and password file <none>
I20260201 03:08:18.316671 32711 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:18.316706 32711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:18.316823 32711 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:18.317667 32711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "bb021763e6df4866b930411f7290cf1f"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "0e64f3800b2beb50643123d258ee0b4f"
server_key_iv: "23aadc88ebd973f167d129580d5f95d2"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.317994 32711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "bb021763e6df4866b930411f7290cf1f"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "0e64f3800b2beb50643123d258ee0b4f"
server_key_iv: "23aadc88ebd973f167d129580d5f95d2"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.319051 32711 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:18.319617 32726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.319764 32711 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:18.319847 32711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "bb021763e6df4866b930411f7290cf1f"
format_stamp: "Formatted at 2026-02-01 03:08:18 on dist-test-slave-ck1n"
server_key: "0e64f3800b2beb50643123d258ee0b4f"
server_key_iv: "23aadc88ebd973f167d129580d5f95d2"
server_key_version: "encryptionkey@0"
I20260201 03:08:18.319912 32711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:18.333040 32711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:18.334839 32711 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:18.334968 32711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:18.335157 32711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:18.335417 32711 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:18.335465 32711 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.335500 32711 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:18.335523 32711 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:18.340684 32711 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:45459
I20260201 03:08:18.340723 371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:45459 every 8 connection(s)
I20260201 03:08:18.341044 32711 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:18.345011 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 32711
I20260201 03:08:18.345114 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:18.345289 29396 external_mini_cluster.cc:1468] Setting key 244ed9aa2101c17a4e1b09f872c42165
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[32327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:18.347858 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.341604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:48283 (local address 127.28.181.62:39395)
0201 03:08:18.341679 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.341681 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.341816 (+ 135us) server_negotiation.cc:408] Connection header received
0201 03:08:18.342238 (+ 422us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.342240 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.342249 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.342295 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.342868 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.343293 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.343739 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.343836 (+ 97us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.346029 (+ 2193us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:18.346032 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:18.346032 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:18.346037 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:18.347007 (+ 970us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.347282 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.347283 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.347283 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.347299 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:18.347445 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:18.347445 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:18.347445 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:18.347511 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:18.347551 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:18.347745 (+ 194us) server_negotiation.cc:300] Negotiation successful
0201 03:08:18.347802 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:08:18.347985 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:18.341562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:39395 (local address 127.28.181.3:48283)
0201 03:08:18.341703 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:18.341726 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:18.342168 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:18.342331 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:18.342333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:18.342560 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:18.342803 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.342806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.343359 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:18.343359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:18.343682 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:18.343683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:18.343767 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:18.344497 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:18.344510 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:18.345961 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:18.347068 (+ 1107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.347072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.347074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.347229 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.347345 (+ 116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:18.347345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:18.347346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:18.347405 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:18.347554 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:18.347557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:18.347666 (+ 109us) client_negotiation.cc:769] Sending connection context
0201 03:08:18.347737 (+ 71us) client_negotiation.cc:240] Negotiation successful
0201 03:08:18.347819 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260201 03:08:18.348390 372 heartbeater.cc:344] Connected to a master server at 127.28.181.62:39395
I20260201 03:08:18.348456 372 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:18.348637 372 heartbeater.cc:507] Master 127.28.181.62:39395 requested a full tablet report, sending...
I20260201 03:08:18.348980 32375 ts_manager.cc:194] Registered new tserver with Master: bb021763e6df4866b930411f7290cf1f (127.28.181.3:45459)
I20260201 03:08:18.349278 32375 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:48283
I20260201 03:08:18.356828 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:08:18.441766 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:18.440419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.28.181.62:39395)
0201 03:08:18.440475 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.440477 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.440679 (+ 202us) server_negotiation.cc:408] Connection header received
0201 03:08:18.440970 (+ 291us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.440971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.440980 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.441028 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.441685 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34490: BlockingRecv error: recv got EOF from 127.0.0.1:34490 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260201 03:08:18.442469 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:18.441909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.28.181.62:39395)
0201 03:08:18.441944 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.441946 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.441954 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:18.442059 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.442059 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.442067 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.442114 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.442409 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34502: BlockingRecv error: recv got EOF from 127.0.0.1:34502 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260201 03:08:18.442975 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:18.442604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.28.181.62:39395)
0201 03:08:18.442637 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.442639 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.442646 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:18.442665 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.442665 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.442679 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.442732 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.442921 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34518: BlockingRecv error: recv got EOF from 127.0.0.1:34518 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260201 03:08:18.444273 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:18.443888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.28.181.62:39395)
0201 03:08:18.443937 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:18.443938 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:18.443946 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:18.443962 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:18.443962 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:18.443968 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:18.444013 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:18.444240 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34534: BlockingRecv error: recv got EOF from 127.0.0.1:34534 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:08:18.498828 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32439
I20260201 03:08:18.502730 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32575
I20260201 03:08:18.506255 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32711
I20260201 03:08:18.509840 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 32343
2026-02-01T03:08:18Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2526 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[395](info): set up 2 sockets
Feb 01 03:08:18 dist-test-slave-ck1n krb5kdc[395](info): commencing operation
krb5kdc: starting...
W20260201 03:08:20.231386 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.702s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:20.335618 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:39925
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:39925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:20.399751 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:20.399901 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:20.399935 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:20.401077 411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:20.401129 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:20.401149 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:20.401167 411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:20.401189 411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:20.402690 411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:39925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:39925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:20.402912 411 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:20.403103 411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:20.405229 417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:20.405236 419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:20.405236 416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:20.405802 411 server_base.cc:1060] running on GCE node
I20260201 03:08:20.406003 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:20.406200 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:20.407335 411 hybrid_clock.cc:648] HybridClock initialized: now 1769915300407164 us; error 189 us; skew 500 ppm
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915300, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:20.409164 411 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:20.409585 411 webserver.cc:492] Webserver started at http://127.28.181.62:32917/ using document root <none> and password file <none>
I20260201 03:08:20.409740 411 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:20.409804 411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:20.409919 411 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:20.410647 411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6"
format_stamp: "Formatted at 2026-02-01 03:08:20 on dist-test-slave-ck1n"
server_key: "94d2264dd1af46e183793c7e52522a36"
server_key_iv: "9ec30bd93f4719a32210f98bd94b6a95"
server_key_version: "encryptionkey@0"
I20260201 03:08:20.410913 411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6"
format_stamp: "Formatted at 2026-02-01 03:08:20 on dist-test-slave-ck1n"
server_key: "94d2264dd1af46e183793c7e52522a36"
server_key_iv: "9ec30bd93f4719a32210f98bd94b6a95"
server_key_version: "encryptionkey@0"
I20260201 03:08:20.411919 411 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:20.412442 426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:20.412576 411 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:20.412647 411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6"
format_stamp: "Formatted at 2026-02-01 03:08:20 on dist-test-slave-ck1n"
server_key: "94d2264dd1af46e183793c7e52522a36"
server_key_iv: "9ec30bd93f4719a32210f98bd94b6a95"
server_key_version: "encryptionkey@0"
I20260201 03:08:20.412709 411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:20.423812 411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:20.425009 411 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:20.425124 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:20.428193 411 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:39925
I20260201 03:08:20.428218 478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:39925 every 8 connection(s)
I20260201 03:08:20.428455 411 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:20.428588 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 411
I20260201 03:08:20.428682 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:20.428835 29396 external_mini_cluster.cc:1468] Setting key bef80c67fb856ccba95316547878001c
I20260201 03:08:20.429015 479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:20.431439 479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6: Bootstrap starting.
I20260201 03:08:20.431893 479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:20.432097 479 log.cc:826] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:20.432677 479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6: No bootstrap required, opened a new log
I20260201 03:08:20.433517 479 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } }
I20260201 03:08:20.433624 479 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:20.433655 479 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb7c182bf1b4f10b41dccb8b413cbc6, State: Initialized, Role: FOLLOWER
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:20.433765 479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } }
I20260201 03:08:20.433902 479 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:20.433946 479 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:20.433990 479 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:20.434652 479 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } }
I20260201 03:08:20.434762 479 leader_election.cc:304] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb7c182bf1b4f10b41dccb8b413cbc6; no voters:
I20260201 03:08:20.434897 479 leader_election.cc:290] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:20.435057 485 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:20.435104 479 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:20.435200 485 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb7c182bf1b4f10b41dccb8b413cbc6, State: Running, Role: LEADER
I20260201 03:08:20.435312 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } }
I20260201 03:08:20.435617 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } } }
I20260201 03:08:20.435693 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:20.435606 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb7c182bf1b4f10b41dccb8b413cbc6. Latest consensus state: current_term: 1 leader_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb7c182bf1b4f10b41dccb8b413cbc6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 39925 } } }
I20260201 03:08:20.435829 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:20.436163 491 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:20.436555 491 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:20.437754 491 catalog_manager.cc:1357] Generated new cluster ID: 0e8478783a9240f1b3553237351097d4
I20260201 03:08:20.437820 491 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:20.437979 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:20.429241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.28.181.62:39925)
0201 03:08:20.429386 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:20.429397 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:20.429418 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:08:20.430300 (+ 882us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:20.430305 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:20.430343 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:20.430459 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:20.431097 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:20.431726 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:20.432249 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:20.432492 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:20.434190 (+ 1698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:20.434197 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:20.434197 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:20.434215 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:20.436477 (+ 2262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:20.436785 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:20.436787 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:20.436787 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:20.436855 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:20.437038 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:20.437039 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:20.437039 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:20.437228 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:20.437363 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:20.437718 (+ 355us) server_negotiation.cc:300] Negotiation successful
0201 03:08:20.437842 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260201 03:08:20.446451 29396 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:08:20.450050 491 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:20.450464 491 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:20.455024 491 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0fb7c182bf1b4f10b41dccb8b413cbc6: Generated new TSK 0
I20260201 03:08:20.455180 491 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:08:20.459672 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:20.457100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.28.181.62:39925)
0201 03:08:20.457128 (+ 28us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:20.457129 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:20.457223 (+ 94us) server_negotiation.cc:408] Connection header received
0201 03:08:20.457277 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:20.457278 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:20.457284 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:20.457327 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:20.457814 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:20.458204 (+ 390us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:20.458853 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:20.458950 (+ 97us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:20.459285 (+ 335us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:20.459288 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:20.459289 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:20.459439 (+ 150us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:20.459450 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:20.459454 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:20.459494 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:20.459568 (+ 74us) server_negotiation.cc:300] Negotiation successful
0201 03:08:20.459586 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260201 03:08:20.461714 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 411
2026-02-01T03:08:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1949 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[512](info): set up 2 sockets
Feb 01 03:08:20 dist-test-slave-ck1n krb5kdc[512](info): commencing operation
krb5kdc: starting...
W20260201 03:08:22.208846 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.731s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/test-user.keytab.
W20260201 03:08:22.915735 29396 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.508s user 0.000s sys 0.001s
Feb 01 03:08:22 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
W20260201 03:08:24.018448 29396 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.28.181.62: real 1.087s user 0.001s sys 0.001s
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
W20260201 03:08:25.445225 29396 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.28.181.62: real 1.427s user 0.001s sys 0.002s
I20260201 03:08:25.445556 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44679
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:44679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:25.522214 528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:25.522387 528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:25.522418 528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:25.523679 528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:25.523736 528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:25.523756 528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:25.523780 528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:25.523805 528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:25.525229 528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:44679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:25.525436 528 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:25.525595 528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:25.527966 534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:25.527973 533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:25.528237 528 server_base.cc:1060] running on GCE node
W20260201 03:08:25.528359 536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:25.528565 528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:25.528770 528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:25.529914 528 hybrid_clock.cc:648] HybridClock initialized: now 1769915305529893 us; error 36 us; skew 500 ppm
Feb 01 03:08:25 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:25.532053 528 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:25.532456 528 webserver.cc:492] Webserver started at http://127.28.181.62:38679/ using document root <none> and password file <none>
I20260201 03:08:25.532583 528 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:25.532610 528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:25.532675 528 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:25.533382 528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "9901736e4bcb48f2a97d39ee955e8f3e"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "832014266b34a7a20bcf8fef5e710d48"
server_key_iv: "7fa4e93d56fd3ed13613f7baaf7c7272"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.533601 528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "9901736e4bcb48f2a97d39ee955e8f3e"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "832014266b34a7a20bcf8fef5e710d48"
server_key_iv: "7fa4e93d56fd3ed13613f7baaf7c7272"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.534636 528 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:25.535279 543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:25.535488 528 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:25.535562 528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "9901736e4bcb48f2a97d39ee955e8f3e"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "832014266b34a7a20bcf8fef5e710d48"
server_key_iv: "7fa4e93d56fd3ed13613f7baaf7c7272"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.535607 528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:25.562366 528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:25.567426 528 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:25.567512 528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:25.571234 528 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:44679
I20260201 03:08:25.571305 595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:44679 every 8 connection(s)
I20260201 03:08:25.571559 528 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:25.572150 596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:25.574493 596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e: Bootstrap starting.
I20260201 03:08:25.575106 596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:25.575362 596 log.cc:826] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:25.576010 596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e: No bootstrap required, opened a new log
I20260201 03:08:25.577169 596 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } }
I20260201 03:08:25.577275 596 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:25.577293 596 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9901736e4bcb48f2a97d39ee955e8f3e, State: Initialized, Role: FOLLOWER
I20260201 03:08:25.577455 596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } }
I20260201 03:08:25.577504 596 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:25.577523 596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:25.577589 596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:25.579612 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 528
I20260201 03:08:25.579704 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:25.579802 29396 external_mini_cluster.cc:1468] Setting key a90a3e0c411e8d8821e5a5c5745b2762
Feb 01 03:08:25 dist-test-slave-ck1n krb5kdc[512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:25.586949 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:25.580184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.28.181.62:44679)
0201 03:08:25.580326 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:25.580337 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:25.580360 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:08:25.580445 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:25.580448 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:25.580479 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:25.580587 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:25.581217 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:25.581894 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:25.582445 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:25.582658 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:25.584469 (+ 1811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:25.584475 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:25.584476 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:25.584489 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:25.585883 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:25.586160 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:25.586162 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:25.586163 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:25.586201 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:25.586317 (+ 116us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:25.586317 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:25.586317 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:25.586479 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:25.586579 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:25.586656 (+ 77us) server_negotiation.cc:300] Negotiation successful
0201 03:08:25.586743 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260201 03:08:25.626245 596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } }
I20260201 03:08:25.626389 596 leader_election.cc:304] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9901736e4bcb48f2a97d39ee955e8f3e; no voters:
I20260201 03:08:25.626533 596 leader_election.cc:290] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:25.626562 601 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:25.626654 601 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [term 1 LEADER]: Becoming Leader. State: Replica: 9901736e4bcb48f2a97d39ee955e8f3e, State: Running, Role: LEADER
I20260201 03:08:25.626744 601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } }
I20260201 03:08:25.626775 596 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:25.627221 603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9901736e4bcb48f2a97d39ee955e8f3e. Latest consensus state: current_term: 1 leader_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } } }
I20260201 03:08:25.627456 603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:25.627389 602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9901736e4bcb48f2a97d39ee955e8f3e" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44679 } } }
I20260201 03:08:25.627687 602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:25.627874 611 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:25.628333 611 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:25.630273 611 catalog_manager.cc:1357] Generated new cluster ID: 1caddc7d3af54620af52867e49dd70e7
I20260201 03:08:25.630331 611 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:25.639755 611 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:25.640193 611 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:25.647007 611 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9901736e4bcb48f2a97d39ee955e8f3e: Generated new TSK 0
I20260201 03:08:25.647156 611 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
W20260201 03:08:25.902077 29396 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.28.181.1: real 0.248s user 0.000s sys 0.002s
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:25.919507 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44679
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:25.986495 624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:25.986632 624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:25.986651 624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:25.987763 624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:25.987795 624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:25.987816 624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:25.989045 624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:25.989178 624 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:25.989282 624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:25.991374 632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:25.991441 624 server_base.cc:1060] running on GCE node
W20260201 03:08:25.991380 630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:25.991379 629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:25.991737 624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:25.991896 624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:25.993019 624 hybrid_clock.cc:648] HybridClock initialized: now 1769915305993004 us; error 46 us; skew 500 ppm
Feb 01 03:08:25 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:25.994910 624 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:25.995365 624 webserver.cc:492] Webserver started at http://127.28.181.1:36875/ using document root <none> and password file <none>
I20260201 03:08:25.995551 624 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:25.995587 624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:25.995711 624 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:25.996476 624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "663a56e5da084939ab543baf62cdf00c"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "82cafaddcd9b38ae42441f6f5e353a70"
server_key_iv: "83cc0b58bab5528b5171839d923f4e28"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.996752 624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "663a56e5da084939ab543baf62cdf00c"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "82cafaddcd9b38ae42441f6f5e353a70"
server_key_iv: "83cc0b58bab5528b5171839d923f4e28"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.997908 624 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:25.998509 639 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:25.998703 624 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:25.998775 624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "663a56e5da084939ab543baf62cdf00c"
format_stamp: "Formatted at 2026-02-01 03:08:25 on dist-test-slave-ck1n"
server_key: "82cafaddcd9b38ae42441f6f5e353a70"
server_key_iv: "83cc0b58bab5528b5171839d923f4e28"
server_key_version: "encryptionkey@0"
I20260201 03:08:25.998842 624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:26.007210 624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:26.008477 624 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:26.008598 624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:26.008790 624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:26.009029 624 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:26.009081 624 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.009115 624 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:26.009138 624 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.014087 624 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:35919
I20260201 03:08:26.014111 752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:35919 every 8 connection(s)
I20260201 03:08:26.014360 624 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:26.021059 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.015013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:39525 (local address 127.28.181.62:44679)
0201 03:08:26.015072 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:26.015074 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:26.015210 (+ 136us) server_negotiation.cc:408] Connection header received
0201 03:08:26.015556 (+ 346us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:26.015557 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:26.015571 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:26.015624 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:26.016204 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.016632 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.017092 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.017188 (+ 96us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.018955 (+ 1767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:26.018960 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:26.018960 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:26.018965 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:26.020226 (+ 1261us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.020488 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.020488 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.020489 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.020504 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.020644 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.020644 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.020644 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.020707 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:26.020744 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:26.020911 (+ 167us) server_negotiation.cc:300] Negotiation successful
0201 03:08:26.020993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:26.021224 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.014980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44679 (local address 127.28.181.1:39525)
0201 03:08:26.015105 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:26.015132 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:26.015492 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:26.015655 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:26.015658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:26.015906 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:26.016143 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.016145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.016718 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.016719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:26.017014 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.017015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.017120 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.017839 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:26.017851 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:26.018879 (+ 1028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:26.020288 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.020292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.020293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.020432 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.020554 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.020554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.020554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.020604 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.020750 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:26.020753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:26.020829 (+ 76us) client_negotiation.cc:769] Sending connection context
0201 03:08:26.020896 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:08:26.020992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260201 03:08:26.021697 753 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44679
I20260201 03:08:26.021829 753 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:26.022020 753 heartbeater.cc:507] Master 127.28.181.62:44679 requested a full tablet report, sending...
I20260201 03:08:26.022432 560 ts_manager.cc:194] Registered new tserver with Master: 663a56e5da084939ab543baf62cdf00c (127.28.181.1:35919)
I20260201 03:08:26.022465 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 624
I20260201 03:08:26.022567 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:26.022718 29396 external_mini_cluster.cc:1468] Setting key a8e0d0f7e7b11284686e3545741f105a
I20260201 03:08:26.023248 560 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:39525
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:26.060236 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44679
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:26.127899 760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:26.128022 760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:26.128041 760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:26.129276 760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:26.129313 760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:26.129334 760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:26.130661 760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:26.130828 760 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:26.130997 760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:26.133050 768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:26.133041 766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:26.133070 760 server_base.cc:1060] running on GCE node
W20260201 03:08:26.133038 765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:26.133340 760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:26.133507 760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:26.134631 760 hybrid_clock.cc:648] HybridClock initialized: now 1769915306134613 us; error 30 us; skew 500 ppm
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:26.136595 760 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:26.137033 760 webserver.cc:492] Webserver started at http://127.28.181.2:46541/ using document root <none> and password file <none>
I20260201 03:08:26.137195 760 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:26.137256 760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:26.137352 760 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:26.138172 760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "605429cd04f6458095e7e9cee765409c"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "ca812de362c16e915b12e7bd3b8d581e"
server_key_iv: "ee0df45d40281aa94452144f54791a82"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.138456 760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "605429cd04f6458095e7e9cee765409c"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "ca812de362c16e915b12e7bd3b8d581e"
server_key_iv: "ee0df45d40281aa94452144f54791a82"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.139497 760 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:26.140101 775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.140271 760 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:26.140350 760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "605429cd04f6458095e7e9cee765409c"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "ca812de362c16e915b12e7bd3b8d581e"
server_key_iv: "ee0df45d40281aa94452144f54791a82"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.140429 760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:26.149452 760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:26.150799 760 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:26.150923 760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:26.151113 760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:26.151355 760 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:26.151405 760 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.151437 760 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:26.151454 760 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.156428 760 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:46013
I20260201 03:08:26.156487 888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:46013 every 8 connection(s)
I20260201 03:08:26.156718 760 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:26.163058 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.157243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:37029 (local address 127.28.181.62:44679)
0201 03:08:26.157294 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:26.157296 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:26.157431 (+ 135us) server_negotiation.cc:408] Connection header received
0201 03:08:26.157747 (+ 316us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:26.157748 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:26.157756 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:26.157841 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:26.158359 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.158763 (+ 404us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.159190 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.159284 (+ 94us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.160957 (+ 1673us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:26.160966 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:26.160967 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:26.160972 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:26.162216 (+ 1244us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.162503 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.162504 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.162504 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.162519 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.162670 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.162671 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.162671 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.162732 (+ 61us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:26.162773 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:26.162941 (+ 168us) server_negotiation.cc:300] Negotiation successful
0201 03:08:26.163005 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260201 03:08:26.163097 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.157216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44679 (local address 127.28.181.2:37029)
0201 03:08:26.157339 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:26.157362 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:26.157687 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:26.157874 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:26.157877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:26.158083 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:26.158295 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.158297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.158833 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.158834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:26.159131 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.159132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.159211 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.159830 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:26.159841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:26.160889 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:26.162279 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.162283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.162284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.162445 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.162570 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.162571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.162571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.162621 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.162779 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:26.162781 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:26.162855 (+ 74us) client_negotiation.cc:769] Sending connection context
0201 03:08:26.162921 (+ 66us) client_negotiation.cc:240] Negotiation successful
0201 03:08:26.162994 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":76,"threads_started":1}
I20260201 03:08:26.163362 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 760
I20260201 03:08:26.163463 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:26.163563 889 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44679
I20260201 03:08:26.163602 29396 external_mini_cluster.cc:1468] Setting key e0ab07c948eb44bb7138cd9711a77234
I20260201 03:08:26.163640 889 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:26.163808 889 heartbeater.cc:507] Master 127.28.181.62:44679 requested a full tablet report, sending...
I20260201 03:08:26.164117 560 ts_manager.cc:194] Registered new tserver with Master: 605429cd04f6458095e7e9cee765409c (127.28.181.2:46013)
I20260201 03:08:26.164549 560 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:37029
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:26.198505 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44679
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:26.263317 896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:26.263477 896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:26.263511 896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:26.264681 896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:26.264746 896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:26.264803 896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:26.266188 896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:26.266399 896 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:26.266559 896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:08:26.268680 896 server_base.cc:1060] running on GCE node
W20260201 03:08:26.268638 901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:26.268644 902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:26.268656 904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:26.269033 896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:26.269240 896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:26.270370 896 hybrid_clock.cc:648] HybridClock initialized: now 1769915306270360 us; error 26 us; skew 500 ppm
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:26.272274 896 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:26.272681 896 webserver.cc:492] Webserver started at http://127.28.181.3:44709/ using document root <none> and password file <none>
I20260201 03:08:26.272855 896 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:26.272912 896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:26.273007 896 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:26.273725 896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "54a0aceb73b04079a18dd9e353bbed23"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "3ec6f7b0d068a90004c77605dfbce728"
server_key_iv: "99d73440710a2ac5d3605ac1cd7bd90e"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.274053 896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "54a0aceb73b04079a18dd9e353bbed23"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "3ec6f7b0d068a90004c77605dfbce728"
server_key_iv: "99d73440710a2ac5d3605ac1cd7bd90e"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.275116 896 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:26.275738 911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.275928 896 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:26.276000 896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "54a0aceb73b04079a18dd9e353bbed23"
format_stamp: "Formatted at 2026-02-01 03:08:26 on dist-test-slave-ck1n"
server_key: "3ec6f7b0d068a90004c77605dfbce728"
server_key_iv: "99d73440710a2ac5d3605ac1cd7bd90e"
server_key_version: "encryptionkey@0"
I20260201 03:08:26.276067 896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:26.289419 896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:26.291143 896 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:26.291281 896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:26.291473 896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:26.291723 896 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:26.291778 896 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.291807 896 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:26.291832 896 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:26.296978 896 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:39201
I20260201 03:08:26.297010 1024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:39201 every 8 connection(s)
I20260201 03:08:26.297232 896 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:26.301630 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 896
I20260201 03:08:26.301697 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:26.301895 29396 external_mini_cluster.cc:1468] Setting key 14ecdd9afa42832a2eed5c2ff596cd02
I20260201 03:08:26.304065 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.297813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:39003 (local address 127.28.181.62:44679)
0201 03:08:26.297853 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:26.297854 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:26.297970 (+ 116us) server_negotiation.cc:408] Connection header received
0201 03:08:26.298290 (+ 320us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:26.298290 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:26.298298 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:26.298348 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:26.298898 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.299382 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.299811 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.299907 (+ 96us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.301590 (+ 1683us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:26.301594 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:26.301594 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:26.301599 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:26.303060 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.303436 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.303437 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.303437 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.303458 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.303645 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.303645 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.303645 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.303714 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:26.303760 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:26.303944 (+ 184us) server_negotiation.cc:300] Negotiation successful
0201 03:08:26.304013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:26.304100 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.297719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44679 (local address 127.28.181.3:39003)
0201 03:08:26.297900 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:26.297919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:26.298224 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:26.298406 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:26.298408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:26.298614 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:26.298832 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.298834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.299450 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.299451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:26.299751 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:26.299752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.299830 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.300434 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:26.300445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:26.301513 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:26.303123 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.303127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.303129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.303278 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.303519 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:26.303519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:26.303519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:26.303589 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:26.303764 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:26.303767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:26.303855 (+ 88us) client_negotiation.cc:769] Sending connection context
0201 03:08:26.303922 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:08:26.303998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":108,"threads_started":1}
I20260201 03:08:26.304495 1025 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44679
I20260201 03:08:26.304558 1025 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:26.304731 1025 heartbeater.cc:507] Master 127.28.181.62:44679 requested a full tablet report, sending...
I20260201 03:08:26.305053 560 ts_manager.cc:194] Registered new tserver with Master: 54a0aceb73b04079a18dd9e353bbed23 (127.28.181.3:39201)
I20260201 03:08:26.305375 560 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:39003
I20260201 03:08:26.313397 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915306, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:26.330312 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:26.325253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.28.181.62:44679)
0201 03:08:26.325284 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:26.325286 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:26.325389 (+ 103us) server_negotiation.cc:408] Connection header received
0201 03:08:26.325419 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:26.325420 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:26.325426 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:26.325471 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:26.326114 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.326467 (+ 353us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:26.326903 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:26.326987 (+ 84us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:26.328649 (+ 1662us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:26.328651 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:26.328652 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:26.328656 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:26.329583 (+ 927us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.329857 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.329858 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.329858 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.329874 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:26.329991 (+ 117us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:26.329991 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:26.329991 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:26.330080 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:26.330140 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:26.330214 (+ 74us) server_negotiation.cc:300] Negotiation successful
0201 03:08:26.330264 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260201 03:08:26.330514 560 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41972
I20260201 03:08:26.330902 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 624
I20260201 03:08:26.335126 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 760
I20260201 03:08:26.339032 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 896
I20260201 03:08:26.342376 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 528
2026-02-01T03:08:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5882 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[1037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[1037](info): set up 2 sockets
Feb 01 03:08:26 dist-test-slave-ck1n krb5kdc[1037](info): commencing operation
krb5kdc: starting...
W20260201 03:08:28.068986 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.708s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:28.204576 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44761
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:44761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:28.270424 1053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:28.270593 1053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:28.270625 1053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:28.271780 1053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:28.271853 1053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:28.271873 1053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:28.271890 1053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:28.271915 1053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:28.273243 1053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:44761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:28.273447 1053 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:28.273614 1053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:28.275719 1061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.275703 1059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.275693 1058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.275990 1053 server_base.cc:1060] running on GCE node
I20260201 03:08:28.276204 1053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:28.276389 1053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:28.277535 1053 hybrid_clock.cc:648] HybridClock initialized: now 1769915308277530 us; error 34 us; skew 500 ppm
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:28.279479 1053 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:28.279963 1053 webserver.cc:492] Webserver started at http://127.28.181.62:38267/ using document root <none> and password file <none>
I20260201 03:08:28.280133 1053 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:28.280185 1053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:28.280289 1053 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:28.281042 1053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "425b2bf644d142c28cb27e14bb4eda03"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4de7598fddf98ff4fb264dfe782d9073"
server_key_iv: "b96f9283598449597221b5449a6a8918"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.281322 1053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "425b2bf644d142c28cb27e14bb4eda03"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4de7598fddf98ff4fb264dfe782d9073"
server_key_iv: "b96f9283598449597221b5449a6a8918"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.282409 1053 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:28.282981 1068 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.283138 1053 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:28.283216 1053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "425b2bf644d142c28cb27e14bb4eda03"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4de7598fddf98ff4fb264dfe782d9073"
server_key_iv: "b96f9283598449597221b5449a6a8918"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.283283 1053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:28.288642 1053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:28.289964 1053 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:28.290082 1053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:28.293491 1053 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:44761
I20260201 03:08:28.293515 1120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:44761 every 8 connection(s)
I20260201 03:08:28.293808 1053 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:28.294287 1121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:28.296396 1121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: Bootstrap starting.
I20260201 03:08:28.296828 1121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:28.297005 1121 log.cc:826] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:28.297497 1121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: No bootstrap required, opened a new log
I20260201 03:08:28.297712 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1053
I20260201 03:08:28.297814 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:28.297940 29396 external_mini_cluster.cc:1468] Setting key 67cd73a5f7d3a5ded10c67d45207ba59
I20260201 03:08:28.298887 1121 raft_consensus.cc:359] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } }
I20260201 03:08:28.299022 1121 raft_consensus.cc:385] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:28.299054 1121 raft_consensus.cc:740] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425b2bf644d142c28cb27e14bb4eda03, State: Initialized, Role: FOLLOWER
I20260201 03:08:28.299175 1121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } }
I20260201 03:08:28.299242 1121 raft_consensus.cc:399] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:28.299268 1121 raft_consensus.cc:493] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:28.299304 1121 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:28.299860 1121 raft_consensus.cc:515] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } }
I20260201 03:08:28.299942 1121 leader_election.cc:304] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 425b2bf644d142c28cb27e14bb4eda03; no voters:
I20260201 03:08:28.300086 1121 leader_election.cc:290] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:28.300130 1126 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:28.300256 1126 raft_consensus.cc:697] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [term 1 LEADER]: Becoming Leader. State: Replica: 425b2bf644d142c28cb27e14bb4eda03, State: Running, Role: LEADER
I20260201 03:08:28.300271 1121 sys_catalog.cc:565] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:28.300401 1126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } }
I20260201 03:08:28.300688 1128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425b2bf644d142c28cb27e14bb4eda03. Latest consensus state: current_term: 1 leader_uuid: "425b2bf644d142c28cb27e14bb4eda03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } } }
I20260201 03:08:28.300750 1128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:28.300982 1127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "425b2bf644d142c28cb27e14bb4eda03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425b2bf644d142c28cb27e14bb4eda03" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44761 } } }
I20260201 03:08:28.301029 1127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03 [sys.catalog]: This master's current role is: LEADER
W20260201 03:08:28.302064 1142 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:28.302265 1142 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:28.306126 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.298350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.28.181.62:44761)
0201 03:08:28.298510 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.298520 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.298535 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:08:28.298861 (+ 326us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.298865 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.298904 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.298995 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.299826 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.300797 (+ 971us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.301579 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.301782 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.303555 (+ 1773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.303562 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.303562 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.303580 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.304955 (+ 1375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.305298 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.305300 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.305300 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.305345 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.305486 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.305486 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.305487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.305635 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.305774 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.305898 (+ 124us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.306009 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260201 03:08:28.309903 1132 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:28.310066 1132 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:28.311175 1132 catalog_manager.cc:1357] Generated new cluster ID: 12976cdce24d4761b825ca06816d6d13
I20260201 03:08:28.311221 1132 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:28.333273 1132 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:28.333912 1132 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:28.338707 1132 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 425b2bf644d142c28cb27e14bb4eda03: Generated new TSK 0
I20260201 03:08:28.338852 1132 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:28.382026 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44761
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:28.449703 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:28.449847 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:28.449867 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:28.451099 1149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:28.451139 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:28.451160 1149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:28.452481 1149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:28.452615 1149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:28.452785 1149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:28.455084 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.455132 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.455103 1149 server_base.cc:1060] running on GCE node
W20260201 03:08:28.455091 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.455440 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:28.455657 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:28.456816 1149 hybrid_clock.cc:648] HybridClock initialized: now 1769915308456806 us; error 23 us; skew 500 ppm
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:28.458820 1149 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:28.459293 1149 webserver.cc:492] Webserver started at http://127.28.181.1:45777/ using document root <none> and password file <none>
I20260201 03:08:28.459419 1149 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:28.459445 1149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:28.459508 1149 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:28.460211 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "843457683aef43a7818d6d7c7cb13d1d"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4323d0f555027d1ff44cbeb7246fe059"
server_key_iv: "f66b12efa65489b22c4bb4b06e73a4ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.460428 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "843457683aef43a7818d6d7c7cb13d1d"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4323d0f555027d1ff44cbeb7246fe059"
server_key_iv: "f66b12efa65489b22c4bb4b06e73a4ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.461434 1149 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260201 03:08:28.462014 1164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.462195 1149 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.462249 1149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "843457683aef43a7818d6d7c7cb13d1d"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4323d0f555027d1ff44cbeb7246fe059"
server_key_iv: "f66b12efa65489b22c4bb4b06e73a4ad"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.462327 1149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:28.477506 1149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:28.478880 1149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:28.479003 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:28.479183 1149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:28.479453 1149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:28.479480 1149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.479498 1149 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:28.479542 1149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.484426 1149 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:39481
I20260201 03:08:28.484470 1277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:39481 every 8 connection(s)
I20260201 03:08:28.484716 1149 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:28.485116 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1149
I20260201 03:08:28.485208 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:28.485400 29396 external_mini_cluster.cc:1468] Setting key 6909fadf7f285735de66949d0e45ca73
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
I20260201 03:08:28.492437 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.485372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:43125 (local address 127.28.181.62:44761)
0201 03:08:28.485414 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.485416 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.485577 (+ 161us) server_negotiation.cc:408] Connection header received
0201 03:08:28.486122 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.486124 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.486136 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.486181 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.486844 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.487278 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.487784 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.487951 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.490114 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.490118 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.490118 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.490124 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.491276 (+ 1152us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.491657 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.491658 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.491659 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.491684 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.491874 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.491875 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.491875 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.491981 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.492051 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.492273 (+ 222us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.492356 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:28.492517 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.485325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44761 (local address 127.28.181.1:43125)
0201 03:08:28.485485 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:28.485512 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:28.486030 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:28.486225 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:28.486229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:28.486526 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:28.486754 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.486756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.487371 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.487372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:28.487701 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.487702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.487837 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.488634 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:28.488647 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:28.490022 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:28.491390 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.491395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.491397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.491571 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.491754 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.491754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.491755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.491810 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.492042 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:28.492047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:28.492154 (+ 107us) client_negotiation.cc:769] Sending connection context
0201 03:08:28.492257 (+ 103us) client_negotiation.cc:240] Negotiation successful
0201 03:08:28.492361 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260201 03:08:28.493031 1278 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44761
I20260201 03:08:28.493178 1278 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:28.493371 1278 heartbeater.cc:507] Master 127.28.181.62:44761 requested a full tablet report, sending...
I20260201 03:08:28.493851 1084 ts_manager.cc:194] Registered new tserver with Master: 843457683aef43a7818d6d7c7cb13d1d (127.28.181.1:39481)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
I20260201 03:08:28.494552 1084 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:43125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:28.522439 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44761
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:28.588601 1285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:28.588732 1285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:28.588752 1285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:28.589882 1285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:28.589913 1285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:28.589936 1285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:28.591193 1285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:28.591342 1285 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:28.591449 1285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:28.593466 1293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.593462 1291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.593552 1285 server_base.cc:1060] running on GCE node
W20260201 03:08:28.593540 1290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.593828 1285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:28.593990 1285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:28.595114 1285 hybrid_clock.cc:648] HybridClock initialized: now 1769915308595106 us; error 23 us; skew 500 ppm
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:28.597196 1285 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:28.597645 1285 webserver.cc:492] Webserver started at http://127.28.181.2:37505/ using document root <none> and password file <none>
I20260201 03:08:28.597877 1285 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:28.597913 1285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:28.598018 1285 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:28.598764 1285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "741d06f400dc4f7696566cb9fe0633e9"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "6fac50a9b3f34cfbd4f34d6983e829db"
server_key_iv: "9151ad1d6a2062233a176e96503277c7"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.599038 1285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "741d06f400dc4f7696566cb9fe0633e9"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "6fac50a9b3f34cfbd4f34d6983e829db"
server_key_iv: "9151ad1d6a2062233a176e96503277c7"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.600034 1285 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:28.600606 1300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.600791 1285 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:28.600847 1285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "741d06f400dc4f7696566cb9fe0633e9"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "6fac50a9b3f34cfbd4f34d6983e829db"
server_key_iv: "9151ad1d6a2062233a176e96503277c7"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.600921 1285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:28.620239 1285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:28.621937 1285 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:28.622022 1285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:28.622150 1285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:28.622412 1285 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:28.622440 1285 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.622495 1285 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:28.622524 1285 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.627416 1285 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:32857
I20260201 03:08:28.627471 1413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:32857 every 8 connection(s)
I20260201 03:08:28.627692 1285 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:28.634364 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.628391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:59861 (local address 127.28.181.62:44761)
0201 03:08:28.628477 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.628479 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.628580 (+ 101us) server_negotiation.cc:408] Connection header received
0201 03:08:28.628966 (+ 386us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.628967 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.628983 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.629030 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.629609 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.630084 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.630541 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.630634 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.632334 (+ 1700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.632338 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.632338 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.632344 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.633500 (+ 1156us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.633814 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.633815 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.633816 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.633833 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.633984 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.633984 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.633985 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.634048 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.634087 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.634254 (+ 167us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.634311 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260201 03:08:28.634460 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.628338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44761 (local address 127.28.181.2:59861)
0201 03:08:28.628499 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:28.628522 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:28.628883 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:28.629064 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:28.629067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:28.629298 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:28.629523 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.629525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.630156 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.630157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:28.630473 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.630474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.630560 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.631166 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:28.631178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:28.632259 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:28.633563 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.633566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.633568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.633715 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.633888 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.633888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.633888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.633936 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.634091 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:28.634094 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:28.634174 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:08:28.634245 (+ 71us) client_negotiation.cc:240] Negotiation successful
0201 03:08:28.634321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":116,"threads_started":1}
I20260201 03:08:28.634940 1414 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44761
I20260201 03:08:28.635010 1414 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:28.635141 1414 heartbeater.cc:507] Master 127.28.181.62:44761 requested a full tablet report, sending...
I20260201 03:08:28.635424 1084 ts_manager.cc:194] Registered new tserver with Master: 741d06f400dc4f7696566cb9fe0633e9 (127.28.181.2:32857)
I20260201 03:08:28.635581 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1285
I20260201 03:08:28.635690 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:28.635857 29396 external_mini_cluster.cc:1468] Setting key 45867a8399d966d1fed96743a9c203f1
I20260201 03:08:28.635912 1084 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:59861
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:28.670069 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:44761
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:28.734762 1421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:28.734890 1421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:28.734912 1421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:28.736285 1421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:28.736325 1421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:28.736346 1421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:28.737644 1421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:28.737792 1421 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:28.737962 1421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:28.739926 1427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.739928 1426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:28.739955 1429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:28.740224 1421 server_base.cc:1060] running on GCE node
I20260201 03:08:28.740356 1421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:28.740535 1421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:28.741679 1421 hybrid_clock.cc:648] HybridClock initialized: now 1769915308741663 us; error 42 us; skew 500 ppm
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:28.743681 1421 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:28.744119 1421 webserver.cc:492] Webserver started at http://127.28.181.3:44987/ using document root <none> and password file <none>
I20260201 03:08:28.744297 1421 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:28.744351 1421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:28.744457 1421 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:28.745218 1421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "56dbb4c890724e3fbddde588891129e7"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4a41fb25cb9bd3b052234cd3dc9bb9d3"
server_key_iv: "a5fe4d8cd3aa49a83922c9458a8503a4"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.745508 1421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "56dbb4c890724e3fbddde588891129e7"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4a41fb25cb9bd3b052234cd3dc9bb9d3"
server_key_iv: "a5fe4d8cd3aa49a83922c9458a8503a4"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.746592 1421 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:28.747164 1436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.747309 1421 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:28.747385 1421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "56dbb4c890724e3fbddde588891129e7"
format_stamp: "Formatted at 2026-02-01 03:08:28 on dist-test-slave-ck1n"
server_key: "4a41fb25cb9bd3b052234cd3dc9bb9d3"
server_key_iv: "a5fe4d8cd3aa49a83922c9458a8503a4"
server_key_version: "encryptionkey@0"
I20260201 03:08:28.747452 1421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:28.764923 1421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:28.766454 1421 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:28.766579 1421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:28.766755 1421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:28.767010 1421 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:28.767060 1421 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.767089 1421 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:28.767112 1421 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:28.771935 1421 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:37877
I20260201 03:08:28.771965 1549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:37877 every 8 connection(s)
I20260201 03:08:28.772212 1421 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:28.773196 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1421
I20260201 03:08:28.773284 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:28.773422 29396 external_mini_cluster.cc:1468] Setting key 606bd10fe1b1f99a780966f9f6b193f9
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:28.779268 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.772806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:44751 (local address 127.28.181.62:44761)
0201 03:08:28.772868 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.772870 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.773002 (+ 132us) server_negotiation.cc:408] Connection header received
0201 03:08:28.773533 (+ 531us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.773535 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.773546 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.773608 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.774493 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.774992 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.775496 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.775599 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.777424 (+ 1825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.777426 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.777427 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.777432 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.778444 (+ 1012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.778725 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.778726 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.778726 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.778742 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.778878 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.778879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.778879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.778940 (+ 61us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.778980 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.779159 (+ 179us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.779215 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:08:28.779352 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.772773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44761 (local address 127.28.181.3:44751)
0201 03:08:28.772896 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:28.772916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:28.773456 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:28.773730 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:28.773736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:28.774111 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:28.774426 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.774428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.775066 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.775067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:28.775428 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:28.775429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.775521 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.776283 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:28.776295 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:28.777354 (+ 1059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:28.778506 (+ 1152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.778511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.778513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.778664 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.778792 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:28.778792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:28.778793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:28.778834 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:28.778990 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:28.778993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:28.779070 (+ 77us) client_negotiation.cc:769] Sending connection context
0201 03:08:28.779138 (+ 68us) client_negotiation.cc:240] Negotiation successful
0201 03:08:28.779213 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":80,"threads_started":1}
I20260201 03:08:28.779738 1550 heartbeater.cc:344] Connected to a master server at 127.28.181.62:44761
I20260201 03:08:28.779851 1550 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:28.780014 1550 heartbeater.cc:507] Master 127.28.181.62:44761 requested a full tablet report, sending...
I20260201 03:08:28.780345 1084 ts_manager.cc:194] Registered new tserver with Master: 56dbb4c890724e3fbddde588891129e7 (127.28.181.3:37877)
I20260201 03:08:28.780678 1084 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:44751
I20260201 03:08:28.785064 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:28.803807 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.797700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.28.181.1:39481)
0201 03:08:28.797881 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.797886 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.797893 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:28.797912 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.797913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.797924 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.798016 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.798585 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.799160 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.799695 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.799895 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.801582 (+ 1687us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.801609 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.801609 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.801615 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.803091 (+ 1476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.803347 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.803348 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.803349 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.803372 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.803480 (+ 108us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.803480 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.803480 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.803543 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.803621 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.803689 (+ 68us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.803743 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:28.810102 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:28.804850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.28.181.62:44761)
0201 03:08:28.804896 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:28.804897 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:28.805002 (+ 105us) server_negotiation.cc:408] Connection header received
0201 03:08:28.805068 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:28.805069 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:28.805078 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:28.805127 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:28.805614 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.806065 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:28.806549 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:28.806708 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:28.808179 (+ 1471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:28.808182 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:28.808183 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:28.808189 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:28.809131 (+ 942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.809435 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.809436 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.809437 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.809455 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:28.809591 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:28.809591 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:28.809592 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:28.809679 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:28.809772 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:28.809953 (+ 181us) server_negotiation.cc:300] Negotiation successful
0201 03:08:28.810019 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260201 03:08:28.810289 1084 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38172
I20260201 03:08:28.810599 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1149
I20260201 03:08:28.815671 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1285
I20260201 03:08:28.819499 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1421
I20260201 03:08:28.822867 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1053
2026-02-01T03:08:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2481 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1562](info): set up 2 sockets
Feb 01 03:08:28 dist-test-slave-ck1n krb5kdc[1562](info): commencing operation
krb5kdc: starting...
W20260201 03:08:30.587038 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.745s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:30 dist-test-slave-ck1n krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:30.696427 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:35563
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:35563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:30.770879 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:30.771064 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:30.771100 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:30.772432 1578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:30.772507 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:30.772532 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:30.772558 1578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:30.772584 1578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:30.774119 1578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:35563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:35563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:30.774354 1578 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:30.774559 1578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:30.776753 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:30.776813 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:30.776754 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:30.776883 1578 server_base.cc:1060] running on GCE node
I20260201 03:08:30.777194 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:30.777454 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:30.778602 1578 hybrid_clock.cc:648] HybridClock initialized: now 1769915310778590 us; error 24 us; skew 500 ppm
Feb 01 03:08:30 dist-test-slave-ck1n krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:30.780802 1578 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:30.781047 1578 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:30.781085 1578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:30.781203 1578 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:30.782091 1578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "a12df8e099ff49f984f13500f0c2f610"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "8105a367ee5571f240b40b0403194397"
server_key_iv: "d8f9ca19687fd91d27600d595c2d27d6"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.782392 1578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "a12df8e099ff49f984f13500f0c2f610"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "8105a367ee5571f240b40b0403194397"
server_key_iv: "d8f9ca19687fd91d27600d595c2d27d6"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.783478 1578 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:30.784134 1592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:30.784336 1578 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:30.784402 1578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "a12df8e099ff49f984f13500f0c2f610"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "8105a367ee5571f240b40b0403194397"
server_key_iv: "d8f9ca19687fd91d27600d595c2d27d6"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.784484 1578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:30.793624 1578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:30.797874 1578 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:30.797955 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:30.801261 1578 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:35563
I20260201 03:08:30.801317 1644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:35563 every 8 connection(s)
I20260201 03:08:30.801509 1578 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:30.802093 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:30.804062 1645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: Bootstrap starting.
I20260201 03:08:30.804586 1645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:30.804827 1645 log.cc:826] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:30.805441 1645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: No bootstrap required, opened a new log
I20260201 03:08:30.806567 1645 raft_consensus.cc:359] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } }
I20260201 03:08:30.806699 1645 raft_consensus.cc:385] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:30.806733 1645 raft_consensus.cc:740] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12df8e099ff49f984f13500f0c2f610, State: Initialized, Role: FOLLOWER
I20260201 03:08:30.806842 1645 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } }
I20260201 03:08:30.806914 1645 raft_consensus.cc:399] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:30.806944 1645 raft_consensus.cc:493] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:30.806989 1645 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:30.807431 1645 raft_consensus.cc:515] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } }
I20260201 03:08:30.807554 1645 leader_election.cc:304] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12df8e099ff49f984f13500f0c2f610; no voters:
I20260201 03:08:30.807698 1645 leader_election.cc:290] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:30.807753 1648 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:30.807893 1645 sys_catalog.cc:565] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:30.807962 1648 raft_consensus.cc:697] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [term 1 LEADER]: Becoming Leader. State: Replica: a12df8e099ff49f984f13500f0c2f610, State: Running, Role: LEADER
I20260201 03:08:30.808097 1648 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } }
I20260201 03:08:30.808410 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12df8e099ff49f984f13500f0c2f610. Latest consensus state: current_term: 1 leader_uuid: "a12df8e099ff49f984f13500f0c2f610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } } }
I20260201 03:08:30.808470 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:30.808436 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12df8e099ff49f984f13500f0c2f610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12df8e099ff49f984f13500f0c2f610" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 35563 } } }
I20260201 03:08:30.808512 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:30.808982 1657 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:30.809391 1657 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:30.809782 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1578
I20260201 03:08:30.809881 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:30.809998 29396 external_mini_cluster.cc:1468] Setting key ab2f894dc47f5bd86a9e212e293369bd
I20260201 03:08:30.811379 1657 catalog_manager.cc:1357] Generated new cluster ID: 085d2fd391d84500a312f43268f6d4f5
I20260201 03:08:30.811426 1657 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:08:30 dist-test-slave-ck1n krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:30.818646 1657 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:30.819227 1657 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:30.819370 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:30.810461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.28.181.62:35563)
0201 03:08:30.810618 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:30.810628 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:30.810646 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:08:30.810730 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:30.810744 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:30.810795 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:30.810912 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:30.811783 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.812769 (+ 986us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:30.813434 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.813637 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:30.815723 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:30.815728 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:30.815728 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:30.815746 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:30.817591 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:30.817944 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:30.817947 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:30.817947 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:30.818001 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:30.818166 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:30.818167 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:30.818167 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:30.818807 (+ 640us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:30.818938 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:30.819059 (+ 121us) server_negotiation.cc:300] Negotiation successful
0201 03:08:30.819244 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260201 03:08:30.825755 1657 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: Generated new TSK 0
I20260201 03:08:30.825904 1657 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:30.872128 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:35563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:30.938395 1673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:30.938513 1673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:30.938534 1673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:30.939714 1673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:30.939776 1673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:30.939802 1673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:30.941322 1673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:35563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:30.941484 1673 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:30.941633 1673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:30.943918 1678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:30.944106 1681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:30.943922 1679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:30.944033 1673 server_base.cc:1060] running on GCE node
I20260201 03:08:30.944367 1673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:30.944604 1673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:30.945737 1673 hybrid_clock.cc:648] HybridClock initialized: now 1769915310945725 us; error 31 us; skew 500 ppm
Feb 01 03:08:30 dist-test-slave-ck1n krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:30.947837 1673 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:30.948057 1673 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:30.948091 1673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:30.948156 1673 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:30.948887 1673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "7a3d90ee41db40f48f9e3228006fff29"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "579608932369b0951f7f285be2e1dcf7"
server_key_iv: "1f2f96d6be5360411ff3ff49173812fe"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.949113 1673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "7a3d90ee41db40f48f9e3228006fff29"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "579608932369b0951f7f285be2e1dcf7"
server_key_iv: "1f2f96d6be5360411ff3ff49173812fe"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.950145 1673 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:30.950685 1687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:30.950975 1673 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:30.951038 1673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "7a3d90ee41db40f48f9e3228006fff29"
format_stamp: "Formatted at 2026-02-01 03:08:30 on dist-test-slave-ck1n"
server_key: "579608932369b0951f7f285be2e1dcf7"
server_key_iv: "1f2f96d6be5360411ff3ff49173812fe"
server_key_version: "encryptionkey@0"
I20260201 03:08:30.951115 1673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:30.970186 1673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:30.971702 1673 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:30.971827 1673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:30.972100 1673 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:30.972369 1673 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:30.972398 1673 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:30.972416 1673 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:30.972466 1673 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:30.978036 1673 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:46809
I20260201 03:08:30.978063 1800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:46809 every 8 connection(s)
I20260201 03:08:30.978255 1673 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:30 dist-test-slave-ck1n krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:30.985281 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1673
I20260201 03:08:30.985416 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:30.985584 29396 external_mini_cluster.cc:1468] Setting key 7dbc22b909439abf35550271c8cbf6dd
I20260201 03:08:30.985828 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:30.978769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:40875 (local address 127.28.181.62:35563)
0201 03:08:30.978826 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:30.978829 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:30.978968 (+ 139us) server_negotiation.cc:408] Connection header received
0201 03:08:30.979370 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:30.979371 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:30.979386 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:30.979439 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:30.980047 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.980462 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:30.981291 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.981449 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:30.983280 (+ 1831us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:30.983283 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:30.983284 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:30.983291 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:30.984542 (+ 1251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:30.984952 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:30.984953 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:30.984954 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:30.984977 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:30.985157 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:30.985158 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:30.985158 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:30.985242 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:30.985307 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:30.985598 (+ 291us) server_negotiation.cc:300] Negotiation successful
0201 03:08:30.985697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:30.985878 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:30.978730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:35563 (local address 127.28.181.1:40875)
0201 03:08:30.978865 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:30.978902 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:30.979290 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:30.979494 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:30.979497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:30.979732 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:30.979959 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:30.979962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.980549 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:30.980553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:30.981206 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:30.981208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:30.981315 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:30.981682 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:30.981697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:30.983191 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:30.984681 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:30.984685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:30.984688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:30.984860 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:30.985040 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:30.985041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:30.985042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:30.985099 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:30.985310 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:30.985314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:30.985466 (+ 152us) client_negotiation.cc:769] Sending connection context
0201 03:08:30.985580 (+ 114us) client_negotiation.cc:240] Negotiation successful
0201 03:08:30.985697 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260201 03:08:30.986449 1801 heartbeater.cc:344] Connected to a master server at 127.28.181.62:35563
I20260201 03:08:30.986532 1801 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:30.986692 1801 heartbeater.cc:507] Master 127.28.181.62:35563 requested a full tablet report, sending...
I20260201 03:08:30.987128 1609 ts_manager.cc:194] Registered new tserver with Master: 7a3d90ee41db40f48f9e3228006fff29 (127.28.181.1:46809)
I20260201 03:08:30.988062 1609 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:40875
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:31.024727 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:35563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:31.095953 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.096086 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.096105 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.097383 1808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.097427 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.097451 1808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:31.098855 1808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:35563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.098994 1808 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.099179 1808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.101204 1813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.101264 1814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.101187 1808 server_base.cc:1060] running on GCE node
W20260201 03:08:31.101183 1816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.101454 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.101599 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.102716 1808 hybrid_clock.cc:648] HybridClock initialized: now 1769915311102718 us; error 47 us; skew 500 ppm
Feb 01 03:08:31 dist-test-slave-ck1n krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:31.104789 1808 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:31.105046 1808 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.105083 1808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.105198 1808 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.106073 1808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "893246c3c44845e1911d6c20900751f2"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "fece479a016a8872caf8c4560724b1d2"
server_key_iv: "097119aae0541ac136cb009dda9a912a"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.106352 1808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "893246c3c44845e1911d6c20900751f2"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "fece479a016a8872caf8c4560724b1d2"
server_key_iv: "097119aae0541ac136cb009dda9a912a"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.107322 1808 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:31.107873 1822 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.108021 1808 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:31.108094 1808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "893246c3c44845e1911d6c20900751f2"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "fece479a016a8872caf8c4560724b1d2"
server_key_iv: "097119aae0541ac136cb009dda9a912a"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.108158 1808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.123853 1808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.125051 1808 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.125211 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.125407 1808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:31.125671 1808 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:31.125721 1808 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.125757 1808 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:31.125823 1808 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.131491 1808 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:45291
I20260201 03:08:31.131520 1935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:45291 every 8 connection(s)
I20260201 03:08:31.131726 1808 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:31 dist-test-slave-ck1n krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:31.138077 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1808
I20260201 03:08:31.138180 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:31.138321 29396 external_mini_cluster.cc:1468] Setting key d4e46db02b40a258e0d2ee7c2d0e9bf8
I20260201 03:08:31.139309 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.132399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:32771 (local address 127.28.181.62:35563)
0201 03:08:31.132485 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.132487 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.132627 (+ 140us) server_negotiation.cc:408] Connection header received
0201 03:08:31.133051 (+ 424us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.133052 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.133063 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.133109 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.133800 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.134260 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.135276 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.135448 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.136946 (+ 1498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.136950 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.136950 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:31.136957 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.137999 (+ 1042us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.138386 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.138387 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.138388 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.138414 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.138779 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.138779 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.138780 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.138849 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.138935 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.139190 (+ 255us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.139255 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:08:31.139474 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.132359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:35563 (local address 127.28.181.2:32771)
0201 03:08:31.132514 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.132542 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.132966 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.133159 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.133161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.133396 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.133699 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.133702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.134349 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.134350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.135174 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.135177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.135310 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.135704 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:31.135718 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.136869 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.138067 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:31.138071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:31.138073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:31.138313 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:31.138536 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:31.138536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:31.138537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:31.138719 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:31.138926 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.138931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.139064 (+ 133us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.139166 (+ 102us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.139287 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260201 03:08:31.139855 1936 heartbeater.cc:344] Connected to a master server at 127.28.181.62:35563
I20260201 03:08:31.139952 1936 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:31.140098 1936 heartbeater.cc:507] Master 127.28.181.62:35563 requested a full tablet report, sending...
I20260201 03:08:31.140408 1609 ts_manager.cc:194] Registered new tserver with Master: 893246c3c44845e1911d6c20900751f2 (127.28.181.2:45291)
I20260201 03:08:31.140744 1609 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:32771
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:31.178752 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:35563
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:31.250860 1943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.251001 1943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.251024 1943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.252449 1943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.252488 1943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.252514 1943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:31.253947 1943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:35563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.254153 1943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.254381 1943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.256558 1951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.256558 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.256702 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.256892 1943 server_base.cc:1060] running on GCE node
I20260201 03:08:31.257035 1943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.257164 1943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.258281 1943 hybrid_clock.cc:648] HybridClock initialized: now 1769915311258263 us; error 33 us; skew 500 ppm
Feb 01 03:08:31 dist-test-slave-ck1n krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:31.260421 1943 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:31.260699 1943 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.260736 1943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.260835 1943 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.261583 1943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "a898e8fff0b347ab820d6929309790cd"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "1489c007b3dce46f32feb7a27d4af7b3"
server_key_iv: "d8122ff8443a228fbdd3b09158d75128"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.261906 1943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "a898e8fff0b347ab820d6929309790cd"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "1489c007b3dce46f32feb7a27d4af7b3"
server_key_iv: "d8122ff8443a228fbdd3b09158d75128"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.262907 1943 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:31.263463 1957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.263612 1943 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:31.263667 1943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "a898e8fff0b347ab820d6929309790cd"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "1489c007b3dce46f32feb7a27d4af7b3"
server_key_iv: "d8122ff8443a228fbdd3b09158d75128"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.263707 1943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.275660 1943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.276970 1943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.277093 1943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.277245 1943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:31.277460 1943 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:31.277488 1943 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.277506 1943 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:31.277519 1943 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.282955 1943 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:39445
I20260201 03:08:31.283043 2070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:39445 every 8 connection(s)
I20260201 03:08:31.283236 1943 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:08:31 dist-test-slave-ck1n krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:31.290100 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.283898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:43137 (local address 127.28.181.62:35563)
0201 03:08:31.283975 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.283977 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.284119 (+ 142us) server_negotiation.cc:408] Connection header received
0201 03:08:31.284546 (+ 427us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.284547 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.284559 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.284610 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.285270 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.285725 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.286613 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.286784 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.288117 (+ 1333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.288121 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.288121 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:31.288127 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.289263 (+ 1136us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.289523 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.289524 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.289524 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.289538 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.289661 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.289661 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.289661 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.289727 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.289808 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.289990 (+ 182us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.290047 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:08:31.290190 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.283824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:35563 (local address 127.28.181.3:43137)
0201 03:08:31.283998 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.284043 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.284451 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.284662 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.284665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.284926 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.285169 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.285171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.285864 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.285864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.286531 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.286532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.286646 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.286983 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:31.286996 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.288044 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.289325 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:31.289328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:31.289329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:31.289469 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:31.289582 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:31.289583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:31.289583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:31.289623 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:31.289816 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.289822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.289904 (+ 82us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.289978 (+ 74us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.290056 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260201 03:08:31.290544 2071 heartbeater.cc:344] Connected to a master server at 127.28.181.62:35563
I20260201 03:08:31.290633 2071 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:31.290791 2071 heartbeater.cc:507] Master 127.28.181.62:35563 requested a full tablet report, sending...
I20260201 03:08:31.291123 1609 ts_manager.cc:194] Registered new tserver with Master: a898e8fff0b347ab820d6929309790cd (127.28.181.3:39445)
I20260201 03:08:31.291456 1609 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:43137
I20260201 03:08:31.292243 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 1943
I20260201 03:08:31.292346 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:31.292495 29396 external_mini_cluster.cc:1468] Setting key 3ea3ea2d99f6ce4518d49d885760dd99
I20260201 03:08:31.293118 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:08:31.298614 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.294238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.28.181.62:35563)
0201 03:08:31.294281 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.294282 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.294341 (+ 59us) server_negotiation.cc:408] Connection header received
0201 03:08:31.294370 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.294370 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.294376 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.294417 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.294996 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.295411 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.295906 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.295999 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.296777 (+ 778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.296779 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.296780 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:31.296783 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.297679 (+ 896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.297985 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.297987 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.297987 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.298019 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:31.298169 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:31.298170 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:31.298170 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:31.298258 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.298353 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.298464 (+ 111us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.298531 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260201 03:08:31.299757 1609 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:31.300803 1609 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:31.304869 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:45291 (local address 127.0.0.1:42112)
0201 03:08:31.302553 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.302577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.302635 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.302887 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.302889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.302891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:31.303096 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.303098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.303972 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.303973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.304650 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.304652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304722 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.304745 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.304780 (+ 35us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.304810 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":46,"threads_started":1}
I20260201 03:08:31.305137 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46809 (local address 127.0.0.1:37552)
0201 03:08:31.302621 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.302631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.302675 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.303100 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.303101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.303102 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:31.303248 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.303249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304304 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.304305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.304914 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.304916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304986 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.304990 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.305021 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.305062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":44,"threads_started":1}
I20260201 03:08:31.305497 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:39445 (local address 127.0.0.1:58908)
0201 03:08:31.302761 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.302768 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.302798 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.303022 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.303023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.303025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:31.303158 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.303159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304185 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.304185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.304866 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.304867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304952 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.304955 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.304992 (+ 37us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.305030 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":64,"threads_started":1}
I20260201 03:08:31.305694 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.28.181.2:45291)
0201 03:08:31.302688 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.302695 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.302733 (+ 38us) server_negotiation.cc:408] Connection header received
0201 03:08:31.302768 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.302770 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.302785 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.302853 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:31.303171 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.303887 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.304757 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.305511 (+ 754us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.305553 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.305583 (+ 30us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.305623 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260201 03:08:31.306039 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.28.181.1:46809)
0201 03:08:31.302914 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.302921 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.302930 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:31.302965 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.302966 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.302982 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.303036 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:31.303308 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304230 (+ 922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.305302 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.305852 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.305895 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.305927 (+ 32us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.305970 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":76,"threads_started":1}
I20260201 03:08:31.306284 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.302774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.28.181.3:39445)
0201 03:08:31.302897 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.302903 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.302911 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:31.302934 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.302936 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.302946 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.303001 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:31.303230 (+ 229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.304114 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.305118 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.306135 (+ 1017us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.306177 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.306202 (+ 25us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.306234 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260201 03:08:31.306258 1870 tablet_service.cc:1505] Processing CreateTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f (DEFAULT_TABLE table=test-table [id=93de63c17e044867a963b765382765de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.306525 1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834b5de9045f40dcbc53f7e0a2b50a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.306742 2005 tablet_service.cc:1505] Processing CreateTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f (DEFAULT_TABLE table=test-table [id=93de63c17e044867a963b765382765de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.306957 2005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834b5de9045f40dcbc53f7e0a2b50a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.308620 2085 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Bootstrap starting.
I20260201 03:08:31.309006 2086 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Bootstrap starting.
I20260201 03:08:31.308980 1735 tablet_service.cc:1505] Processing CreateTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f (DEFAULT_TABLE table=test-table [id=93de63c17e044867a963b765382765de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.309242 1735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834b5de9045f40dcbc53f7e0a2b50a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.309382 2085 tablet_bootstrap.cc:654] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.309540 2086 tablet_bootstrap.cc:654] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.309790 2085 log.cc:826] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.309847 2086 log.cc:826] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.310344 2086 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: No bootstrap required, opened a new log
I20260201 03:08:31.310417 2086 ts_tablet_manager.cc:1403] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:08:31.310420 2085 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: No bootstrap required, opened a new log
I20260201 03:08:31.310468 2085 ts_tablet_manager.cc:1403] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:31.311484 2086 raft_consensus.cc:359] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.311599 2086 raft_consensus.cc:385] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.311630 2086 raft_consensus.cc:740] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a898e8fff0b347ab820d6929309790cd, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.311717 2086 consensus_queue.cc:260] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.311972 2071 heartbeater.cc:499] Master 127.28.181.62:35563 was elected leader, sending a full tablet report...
I20260201 03:08:31.312003 2085 raft_consensus.cc:359] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.312094 2086 ts_tablet_manager.cc:1434] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:31.312111 2085 raft_consensus.cc:385] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.312135 2085 raft_consensus.cc:740] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893246c3c44845e1911d6c20900751f2, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.312213 2085 consensus_queue.cc:260] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.312443 2085 ts_tablet_manager.cc:1434] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:31.312458 1936 heartbeater.cc:499] Master 127.28.181.62:35563 was elected leader, sending a full tablet report...
I20260201 03:08:31.312708 2089 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: Bootstrap starting.
I20260201 03:08:31.313304 2089 tablet_bootstrap.cc:654] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.313561 2089 log.cc:826] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.314344 2089 tablet_bootstrap.cc:492] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: No bootstrap required, opened a new log
I20260201 03:08:31.314393 2089 ts_tablet_manager.cc:1403] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:31.315553 2089 raft_consensus.cc:359] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.315649 2089 raft_consensus.cc:385] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.315666 2089 raft_consensus.cc:740] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3d90ee41db40f48f9e3228006fff29, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.315773 2089 consensus_queue.cc:260] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.316017 2089 ts_tablet_manager.cc:1434] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:31.316030 1801 heartbeater.cc:499] Master 127.28.181.62:35563 was elected leader, sending a full tablet report...
W20260201 03:08:31.382174 1937 tablet.cc:2378] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:31.394956 2090 raft_consensus.cc:493] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:31.395035 2090 raft_consensus.cc:515] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.395395 2090 leader_election.cc:290] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 893246c3c44845e1911d6c20900751f2 (127.28.181.2:45291), 7a3d90ee41db40f48f9e3228006fff29 (127.28.181.1:46809)
I20260201 03:08:31.397876 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.395523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:45291 (local address 127.28.181.3:55471)
0201 03:08:31.395560 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.395571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.395614 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.395779 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.395780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.395782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:31.396024 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.396025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.396866 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.396867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.397664 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.397665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.397730 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.397732 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.397757 (+ 25us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.397805 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260201 03:08:31.397886 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.395551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46809 (local address 127.28.181.3:52007)
0201 03:08:31.395695 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.395703 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.395791 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.395912 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.395913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.395914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:31.396147 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.396148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.396873 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.396873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:31.397663 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:31.397664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.397729 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.397732 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.397763 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.397808 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260201 03:08:31.398294 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.395589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:55471 (local address 127.28.181.2:45291)
0201 03:08:31.395629 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.395631 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.395640 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:31.395669 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.395670 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.395680 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.395761 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:31.396101 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.396774 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.397795 (+ 1021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.398184 (+ 389us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.398191 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.398210 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.398236 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:31.398355 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.395630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:52007 (local address 127.28.181.1:46809)
0201 03:08:31.395667 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.395669 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.395810 (+ 141us) server_negotiation.cc:408] Connection header received
0201 03:08:31.395843 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.395844 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.395851 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.395890 (+ 39us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:31.396213 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.396812 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.397756 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.398250 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.398255 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.398272 (+ 17us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.398296 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:31.398573 1755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "834b5de9045f40dcbc53f7e0a2b50a8f" candidate_uuid: "a898e8fff0b347ab820d6929309790cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3d90ee41db40f48f9e3228006fff29" is_pre_election: true
I20260201 03:08:31.398577 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "834b5de9045f40dcbc53f7e0a2b50a8f" candidate_uuid: "a898e8fff0b347ab820d6929309790cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893246c3c44845e1911d6c20900751f2" is_pre_election: true
I20260201 03:08:31.398715 1755 raft_consensus.cc:2468] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a898e8fff0b347ab820d6929309790cd in term 0.
I20260201 03:08:31.398723 1890 raft_consensus.cc:2468] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a898e8fff0b347ab820d6929309790cd in term 0.
I20260201 03:08:31.398921 1960 leader_election.cc:304] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a3d90ee41db40f48f9e3228006fff29, a898e8fff0b347ab820d6929309790cd; no voters:
I20260201 03:08:31.399034 2090 raft_consensus.cc:2804] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:31.399088 2090 raft_consensus.cc:493] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:31.399117 2090 raft_consensus.cc:3060] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:31.399626 2090 raft_consensus.cc:515] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.399765 2090 leader_election.cc:290] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [CANDIDATE]: Term 1 election: Requested vote from peers 893246c3c44845e1911d6c20900751f2 (127.28.181.2:45291), 7a3d90ee41db40f48f9e3228006fff29 (127.28.181.1:46809)
I20260201 03:08:31.399927 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "834b5de9045f40dcbc53f7e0a2b50a8f" candidate_uuid: "a898e8fff0b347ab820d6929309790cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893246c3c44845e1911d6c20900751f2"
I20260201 03:08:31.399995 1890 raft_consensus.cc:3060] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:31.399968 1755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "834b5de9045f40dcbc53f7e0a2b50a8f" candidate_uuid: "a898e8fff0b347ab820d6929309790cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3d90ee41db40f48f9e3228006fff29"
I20260201 03:08:31.400055 1755 raft_consensus.cc:3060] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:31.400584 1890 raft_consensus.cc:2468] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a898e8fff0b347ab820d6929309790cd in term 1.
I20260201 03:08:31.400660 1755 raft_consensus.cc:2468] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a898e8fff0b347ab820d6929309790cd in term 1.
I20260201 03:08:31.400772 1961 leader_election.cc:304] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 893246c3c44845e1911d6c20900751f2, a898e8fff0b347ab820d6929309790cd; no voters:
I20260201 03:08:31.400861 2090 raft_consensus.cc:2804] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:31.400960 2090 raft_consensus.cc:697] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 1 LEADER]: Becoming Leader. State: Replica: a898e8fff0b347ab820d6929309790cd, State: Running, Role: LEADER
I20260201 03:08:31.401047 2090 consensus_queue.cc:237] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } }
I20260201 03:08:31.401705 1608 catalog_manager.cc:5654] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd reported cstate change: term changed from 0 to 1, leader changed from <none> to a898e8fff0b347ab820d6929309790cd (127.28.181.3). New cstate: current_term: 1 leader_uuid: "a898e8fff0b347ab820d6929309790cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a898e8fff0b347ab820d6929309790cd" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 39445 } health_report { overall_health: HEALTHY } } }
I20260201 03:08:31.420553 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.418802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.28.181.3:39445)
0201 03:08:31.418853 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.418856 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.418864 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:31.418928 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.418929 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.418942 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.418997 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:31.419195 (+ 198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.419576 (+ 381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:31.420089 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:31.420222 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:31.420248 (+ 26us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:31.420314 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:31.420395 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.420450 (+ 55us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.420468 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:08:31.421890 1755 raft_consensus.cc:1275] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 1 FOLLOWER]: Refusing update from remote peer a898e8fff0b347ab820d6929309790cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:31.421895 1890 raft_consensus.cc:1275] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 1 FOLLOWER]: Refusing update from remote peer a898e8fff0b347ab820d6929309790cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:31.422143 2095 consensus_queue.cc:1048] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "893246c3c44845e1911d6c20900751f2" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:31.422235 2090 consensus_queue.cc:1048] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3d90ee41db40f48f9e3228006fff29" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:31.425269 2099 mvcc.cc:204] Tried to move back new op lower bound from 7249573115581997056 to 7249573115499155456. Current Snapshot: MvccSnapshot[applied={T|T < 7249573115581997056 or (T in {7249573115581997056})}]
I20260201 03:08:31.426120 1609 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:31.426203 1609 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:31.426826 1609 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a12df8e099ff49f984f13500f0c2f610: Sending DeleteTablet for 3 replicas of tablet 834b5de9045f40dcbc53f7e0a2b50a8f
I20260201 03:08:31.427114 1870 tablet_service.cc:1552] Processing DeleteTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:31 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:42112
I20260201 03:08:31.427101 2005 tablet_service.cc:1552] Processing DeleteTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:31 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:58908
I20260201 03:08:31.427125 1735 tablet_service.cc:1552] Processing DeleteTablet for tablet 834b5de9045f40dcbc53f7e0a2b50a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:31 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:37552
I20260201 03:08:31.427305 2110 tablet_replica.cc:333] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29: stopping tablet replica
I20260201 03:08:31.427381 2110 raft_consensus.cc:2243] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:31.427395 2109 tablet_replica.cc:333] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: stopping tablet replica
I20260201 03:08:31.427464 2110 raft_consensus.cc:2272] T 834b5de9045f40dcbc53f7e0a2b50a8f P 7a3d90ee41db40f48f9e3228006fff29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:31.427546 2109 raft_consensus.cc:2243] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:31.427567 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1673
I20260201 03:08:31.427696 2109 raft_consensus.cc:2272] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:31.428033 2109 ts_tablet_manager.cc:1916] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:31.429016 2108 tablet_replica.cc:333] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: stopping tablet replica
I20260201 03:08:31.429107 2108 raft_consensus.cc:2243] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:31.429203 2108 raft_consensus.cc:2272] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:31.429307 2109 ts_tablet_manager.cc:1929] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:31.429387 2109 log.cc:1199] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/834b5de9045f40dcbc53f7e0a2b50a8f
I20260201 03:08:31.429647 2109 ts_tablet_manager.cc:1950] T 834b5de9045f40dcbc53f7e0a2b50a8f P 893246c3c44845e1911d6c20900751f2: Deleting consensus metadata
I20260201 03:08:31.434079 1596 catalog_manager.cc:4985] TS 893246c3c44845e1911d6c20900751f2 (127.28.181.2:45291): tablet 834b5de9045f40dcbc53f7e0a2b50a8f (table test-table [id=93de63c17e044867a963b765382765de]) successfully deleted
W20260201 03:08:31.434165 1595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.181.1:46809 (error 108)
I20260201 03:08:31.434471 2108 ts_tablet_manager.cc:1916] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:31.434684 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.434570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46809 (local address 127.0.0.1:37554)
0201 03:08:31.434608 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.434653 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260201 03:08:31.434782 1595 catalog_manager.cc:4712] TS 7a3d90ee41db40f48f9e3228006fff29 (127.28.181.1:46809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 834b5de9045f40dcbc53f7e0a2b50a8f: Network error: Client connection negotiation failed: client connection to 127.28.181.1:46809: connect: Connection refused (error 111)
I20260201 03:08:31.435098 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1808
I20260201 03:08:31.435938 2108 ts_tablet_manager.cc:1929] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:31.436013 2108 log.cc:1199] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/834b5de9045f40dcbc53f7e0a2b50a8f
I20260201 03:08:31.436221 2108 ts_tablet_manager.cc:1950] T 834b5de9045f40dcbc53f7e0a2b50a8f P a898e8fff0b347ab820d6929309790cd: Deleting consensus metadata
I20260201 03:08:31.436650 1596 catalog_manager.cc:4985] TS a898e8fff0b347ab820d6929309790cd (127.28.181.3:39445): tablet 834b5de9045f40dcbc53f7e0a2b50a8f (table test-table [id=93de63c17e044867a963b765382765de]) successfully deleted
I20260201 03:08:31.440419 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1943
I20260201 03:08:31.445143 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 1578
2026-02-01T03:08:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2622 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-02-01T03:08:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:31Z Disabled control of system clock
I20260201 03:08:31.469348 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:43235
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260201 03:08:31.534952 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.535077 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.535096 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.536285 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:31.536329 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.536339 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.536350 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:31.536360 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:31.537554 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:43235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.537678 2115 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.537878 2115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.539772 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.539757 2115 server_base.cc:1060] running on GCE node
W20260201 03:08:31.539758 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.539744 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.540133 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.540323 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.541472 2115 hybrid_clock.cc:648] HybridClock initialized: now 1769915311541469 us; error 36 us; skew 500 ppm
I20260201 03:08:31.542429 2115 webserver.cc:492] Webserver started at http://127.28.181.62:41593/ using document root <none> and password file <none>
I20260201 03:08:31.542601 2115 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.542634 2115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.542752 2115 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.543489 2115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "860f38a6b0484818b0b491d124ad610a"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "dfe5c1e68ff7175c2a0f890a8c5ce9d2"
server_key_iv: "65adc65dc88c36b266f1ee3e9e4c0bde"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.543777 2115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "860f38a6b0484818b0b491d124ad610a"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "dfe5c1e68ff7175c2a0f890a8c5ce9d2"
server_key_iv: "65adc65dc88c36b266f1ee3e9e4c0bde"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.544826 2115 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:31.545588 2129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.545748 2115 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:31.545850 2115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "860f38a6b0484818b0b491d124ad610a"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "dfe5c1e68ff7175c2a0f890a8c5ce9d2"
server_key_iv: "65adc65dc88c36b266f1ee3e9e4c0bde"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.545905 2115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.548555 2115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.548738 2115 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.548837 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.552232 2115 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:43235
I20260201 03:08:31.552254 2181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:43235 every 8 connection(s)
I20260201 03:08:31.552488 2115 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:31.552983 2182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.554913 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: Bootstrap starting.
I20260201 03:08:31.555414 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.555639 2182 log.cc:826] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.556196 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: No bootstrap required, opened a new log
I20260201 03:08:31.557176 2182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } }
I20260201 03:08:31.557268 2182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.557286 2182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860f38a6b0484818b0b491d124ad610a, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.557404 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } }
I20260201 03:08:31.557448 2182 raft_consensus.cc:399] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:31.557497 2182 raft_consensus.cc:493] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:31.557538 2182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:31.558059 2182 raft_consensus.cc:515] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } }
I20260201 03:08:31.558180 2182 leader_election.cc:304] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 860f38a6b0484818b0b491d124ad610a; no voters:
I20260201 03:08:31.558331 2182 leader_election.cc:290] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:31.558380 2185 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:31.558555 2182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:31.558539 2185 raft_consensus.cc:697] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [term 1 LEADER]: Becoming Leader. State: Replica: 860f38a6b0484818b0b491d124ad610a, State: Running, Role: LEADER
I20260201 03:08:31.558714 2185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } }
I20260201 03:08:31.559090 2187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860f38a6b0484818b0b491d124ad610a. Latest consensus state: current_term: 1 leader_uuid: "860f38a6b0484818b0b491d124ad610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } } }
I20260201 03:08:31.559177 2187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:31.559540 2193 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:31.559978 2193 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:31.559326 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "860f38a6b0484818b0b491d124ad610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860f38a6b0484818b0b491d124ad610a" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 43235 } } }
I20260201 03:08:31.560526 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:31.561455 2193 catalog_manager.cc:1357] Generated new cluster ID: 63a0b13812f84cd79640852479662f23
I20260201 03:08:31.561525 2193 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:31.562165 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2115
I20260201 03:08:31.562258 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:31.562397 29396 external_mini_cluster.cc:1468] Setting key f5cfebcca5dd3d760025a320a676c3f8
I20260201 03:08:31.564218 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.562752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.28.181.62:43235)
0201 03:08:31.562933 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.562939 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.562960 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:08:31.563043 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.563048 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.563077 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.563171 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.563761 (+ 590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.563763 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.563764 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.563767 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.563797 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.563803 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.563875 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.563944 (+ 69us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.564098 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260201 03:08:31.574407 2193 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:31.574519 2193 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:31.581146 2193 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: Generated new TSK 0
I20260201 03:08:31.581279 2193 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:08:31.588994 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--builtin_ntp_servers=127.28.181.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260201 03:08:31.657917 2206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.658074 2206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.658102 2206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.659318 2206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.659370 2206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.659404 2206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:31.660825 2206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.661049 2206 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.661245 2206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.663520 2212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.663524 2211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.663745 2206 server_base.cc:1060] running on GCE node
W20260201 03:08:31.663566 2214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.663923 2206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.664114 2206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.665256 2206 hybrid_clock.cc:648] HybridClock initialized: now 1769915311665242 us; error 28 us; skew 500 ppm
I20260201 03:08:31.666296 2206 webserver.cc:492] Webserver started at http://127.28.181.1:43651/ using document root <none> and password file <none>
I20260201 03:08:31.666487 2206 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.666540 2206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.666648 2206 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.667420 2206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "bbf9a89bf9de424c92f9464339ef3baa"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "995e314ccb00679e4fd3ed7f9f48c4da"
server_key_iv: "dcea3e607488b28adb162b99ba77a7a1"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.667693 2206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf9a89bf9de424c92f9464339ef3baa"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "995e314ccb00679e4fd3ed7f9f48c4da"
server_key_iv: "dcea3e607488b28adb162b99ba77a7a1"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.668704 2206 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:31.669324 2220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.669553 2206 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:31.669626 2206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "bbf9a89bf9de424c92f9464339ef3baa"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "995e314ccb00679e4fd3ed7f9f48c4da"
server_key_iv: "dcea3e607488b28adb162b99ba77a7a1"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.669687 2206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.673738 2206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.673921 2206 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.674015 2206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.674183 2206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:31.674460 2206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:31.674501 2206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.674525 2206 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:31.674548 2206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.680716 2206 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:35305
I20260201 03:08:31.680744 2333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:35305 every 8 connection(s)
I20260201 03:08:31.680994 2206 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:31.682610 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2206
I20260201 03:08:31.682703 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:31.682868 29396 external_mini_cluster.cc:1468] Setting key b3741b66e12a4db465f9c755b562eef0
I20260201 03:08:31.683362 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.681524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43235 (local address 127.28.181.1:45415)
0201 03:08:31.681682 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.681706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.682118 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.682319 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.682322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.682329 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.682826 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.682827 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.682849 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.682869 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.683080 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.683085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.683088 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.683171 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.683213 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260201 03:08:31.683451 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.681561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:45415 (local address 127.28.181.62:43235)
0201 03:08:31.681611 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.681613 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.681801 (+ 188us) server_negotiation.cc:408] Connection header received
0201 03:08:31.682207 (+ 406us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.682209 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.682215 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.682269 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.682984 (+ 715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.682986 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.682986 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.682990 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.683006 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.683010 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.683067 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.683375 (+ 308us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.683390 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:08:31.683743 2334 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43235
I20260201 03:08:31.683784 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--builtin_ntp_servers=127.28.181.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260201 03:08:31.683812 2334 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:31.683917 2334 heartbeater.cc:507] Master 127.28.181.62:43235 requested a full tablet report, sending...
I20260201 03:08:31.684342 2146 ts_manager.cc:194] Registered new tserver with Master: bbf9a89bf9de424c92f9464339ef3baa (127.28.181.1:35305)
W20260201 03:08:31.751119 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.751258 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.751278 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.752460 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.752490 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.752511 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:31.753793 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.753931 2337 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.754087 2337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.756340 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.756340 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.756342 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.756654 2337 server_base.cc:1060] running on GCE node
I20260201 03:08:31.756814 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.757021 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.758152 2337 hybrid_clock.cc:648] HybridClock initialized: now 1769915311758145 us; error 24 us; skew 500 ppm
I20260201 03:08:31.759042 2337 webserver.cc:492] Webserver started at http://127.28.181.2:43945/ using document root <none> and password file <none>
I20260201 03:08:31.759212 2337 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.759266 2337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.759371 2337 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.760236 2337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "c1e4fc301c1244caa5f91e762b4f5ae7"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "f10f5079639d5763d1f880171cdaa962"
server_key_iv: "854a99b208ca49976cbea8d0d26b8df6"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.760507 2337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "c1e4fc301c1244caa5f91e762b4f5ae7"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "f10f5079639d5763d1f880171cdaa962"
server_key_iv: "854a99b208ca49976cbea8d0d26b8df6"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.761580 2337 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260201 03:08:31.762308 2351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.762454 2337 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.762528 2337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "c1e4fc301c1244caa5f91e762b4f5ae7"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "f10f5079639d5763d1f880171cdaa962"
server_key_iv: "854a99b208ca49976cbea8d0d26b8df6"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.762595 2337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.765492 2337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.765661 2337 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.765813 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.765996 2337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:31.766274 2337 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:31.766328 2337 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.766367 2337 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:31.766392 2337 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.771770 2337 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:33671
I20260201 03:08:31.771803 2464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:33671 every 8 connection(s)
I20260201 03:08:31.772058 2337 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:31.774348 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.772705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:37773 (local address 127.28.181.62:43235)
0201 03:08:31.772788 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.772790 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.772928 (+ 138us) server_negotiation.cc:408] Connection header received
0201 03:08:31.773330 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.773331 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.773339 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.773399 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.774130 (+ 731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.774132 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.774132 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.774136 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.774152 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.774155 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.774204 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.774280 (+ 76us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.774294 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:08:31.774469 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.772623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43235 (local address 127.28.181.2:37773)
0201 03:08:31.772836 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.772857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.773237 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.773459 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.773462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.773470 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.774020 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.774020 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.774036 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.774053 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.774215 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.774219 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.774220 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.774288 (+ 68us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.774326 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260201 03:08:31.774940 2465 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43235
I20260201 03:08:31.775012 2465 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:31.775095 2465 heartbeater.cc:507] Master 127.28.181.62:43235 requested a full tablet report, sending...
I20260201 03:08:31.775429 2146 ts_manager.cc:194] Registered new tserver with Master: c1e4fc301c1244caa5f91e762b4f5ae7 (127.28.181.2:33671)
I20260201 03:08:31.777288 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2337
I20260201 03:08:31.777384 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:31.777527 29396 external_mini_cluster.cc:1468] Setting key db257a5349b77d49fbd2aa3d36f08348
I20260201 03:08:31.778276 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--builtin_ntp_servers=127.28.181.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260201 03:08:31.845419 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:31.845547 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:31.845566 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:31.846855 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:31.846892 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:31.846913 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:31.848183 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:31.848316 2468 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:31.848538 2468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:31.853883 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:31.854156 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.854743 2468 server_base.cc:1060] running on GCE node
W20260201 03:08:31.854794 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:31.854959 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:31.855134 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:31.856264 2468 hybrid_clock.cc:648] HybridClock initialized: now 1769915311856240 us; error 41 us; skew 500 ppm
I20260201 03:08:31.857203 2468 webserver.cc:492] Webserver started at http://127.28.181.3:36363/ using document root <none> and password file <none>
I20260201 03:08:31.857372 2468 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:31.857407 2468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:31.857522 2468 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:31.858350 2468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "82027ff8aabf4fbba955813b63c1ed57"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "baab1958321168bf9d0161ddc8c0a877"
server_key_iv: "ca4c9e6d2a03dd43098e8065652977ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.858609 2468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "82027ff8aabf4fbba955813b63c1ed57"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "baab1958321168bf9d0161ddc8c0a877"
server_key_iv: "ca4c9e6d2a03dd43098e8065652977ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.859596 2468 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:31.860276 2482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.860476 2468 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:31.860545 2468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "82027ff8aabf4fbba955813b63c1ed57"
format_stamp: "Formatted at 2026-02-01 03:08:31 on dist-test-slave-ck1n"
server_key: "baab1958321168bf9d0161ddc8c0a877"
server_key_iv: "ca4c9e6d2a03dd43098e8065652977ae"
server_key_version: "encryptionkey@0"
I20260201 03:08:31.860613 2468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:31.863317 2468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:31.863497 2468 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:31.863596 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:31.863773 2468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:31.864023 2468 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:31.864053 2468 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.864096 2468 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:31.864122 2468 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:31.868835 2468 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:34713
I20260201 03:08:31.868897 2595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:34713 every 8 connection(s)
I20260201 03:08:31.869127 2468 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:31.871160 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.869689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:35123 (local address 127.28.181.62:43235)
0201 03:08:31.869766 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.869768 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.869925 (+ 157us) server_negotiation.cc:408] Connection header received
0201 03:08:31.870275 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.870276 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.870282 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.870335 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.870945 (+ 610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.870946 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.870947 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.870951 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.870964 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.870967 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.871017 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.871083 (+ 66us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.871100 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:31.871281 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.869668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:43235 (local address 127.28.181.3:35123)
0201 03:08:31.869838 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.869857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.870191 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.870385 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.870388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.870394 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.870837 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.870838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.870852 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.870866 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.871016 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.871020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.871022 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.871101 (+ 79us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.871140 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260201 03:08:31.871641 2596 heartbeater.cc:344] Connected to a master server at 127.28.181.62:43235
I20260201 03:08:31.871701 2596 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:31.871842 2596 heartbeater.cc:507] Master 127.28.181.62:43235 requested a full tablet report, sending...
I20260201 03:08:31.872167 2146 ts_manager.cc:194] Registered new tserver with Master: 82027ff8aabf4fbba955813b63c1ed57 (127.28.181.3:34713)
I20260201 03:08:31.874213 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2468
I20260201 03:08:31.874297 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:31.874446 29396 external_mini_cluster.cc:1468] Setting key 90813372183b4295b72b4bf7e2ea825d
I20260201 03:08:31.875283 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:08:31.877108 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.876360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.28.181.62:43235)
0201 03:08:31.876417 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.876418 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.876503 (+ 85us) server_negotiation.cc:408] Connection header received
0201 03:08:31.876533 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.876533 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.876537 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.876577 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.876972 (+ 395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.876973 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.876973 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.876976 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.876985 (+ 9us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.876987 (+ 2us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.877029 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.877059 (+ 30us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.877068 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:08:31.878312 2146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:31.879364 2146 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:31.882849 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33671 (local address 127.0.0.1:57232)
0201 03:08:31.881433 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.881444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.881490 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.881756 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.881757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.881759 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.882312 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.882313 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.882333 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.882353 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.882623 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.882627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.882630 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.882671 (+ 41us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.882685 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260201 03:08:31.883070 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:34713 (local address 127.0.0.1:60578)
0201 03:08:31.881361 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.881390 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.881443 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.881748 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.881753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.881758 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.882390 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.882390 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.882397 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.882409 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.882621 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.882627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.882629 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.882687 (+ 58us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.882702 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":51,"threads_started":1}
I20260201 03:08:31.883096 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.28.181.2:33671)
0201 03:08:31.881588 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.881595 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.881602 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:31.881622 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.881623 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.881649 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.881988 (+ 339us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.882490 (+ 502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.882492 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.882493 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.882496 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.882532 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.882538 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.882778 (+ 240us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.882804 (+ 26us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.882814 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260201 03:08:31.883196 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:35305 (local address 127.0.0.1:53418)
0201 03:08:31.881685 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:31.881694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:31.881731 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:31.882051 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:31.882051 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:31.882053 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:31.882555 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:31.882556 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:31.882561 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:31.882569 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:31.883079 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:31.883079 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:31.883080 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:08:31.883110 (+ 30us) client_negotiation.cc:240] Negotiation successful
0201 03:08:31.883122 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":89,"threads_started":1}
I20260201 03:08:31.883496 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.28.181.1:35305)
0201 03:08:31.881895 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.881902 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.881912 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:31.881945 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.881947 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.881955 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.882171 (+ 216us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.882968 (+ 797us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.882970 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.882971 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.882975 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.882997 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.883003 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.883354 (+ 351us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.883378 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.883396 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
I20260201 03:08:31.883538 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:31.881321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.28.181.3:34713)
0201 03:08:31.881593 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:31.881601 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:31.881611 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:31.881631 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:31.881633 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:31.881639 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:31.882026 (+ 387us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:31.882498 (+ 472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:31.882502 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:31.882503 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:31.882507 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:31.882540 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:31.882546 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:31.883228 (+ 682us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:31.883251 (+ 23us) server_negotiation.cc:300] Negotiation successful
0201 03:08:31.883265 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":72,"threads_started":1}
I20260201 03:08:31.883625 2399 tablet_service.cc:1505] Processing CreateTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 (DEFAULT_TABLE table=test-table [id=a916231ed91747de9dc2ffb1978313f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.883913 2399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5d883f494049e6a045aacc9f4c52d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.883970 2530 tablet_service.cc:1505] Processing CreateTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 (DEFAULT_TABLE table=test-table [id=a916231ed91747de9dc2ffb1978313f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.883971 2268 tablet_service.cc:1505] Processing CreateTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 (DEFAULT_TABLE table=test-table [id=a916231ed91747de9dc2ffb1978313f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:31.884209 2530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5d883f494049e6a045aacc9f4c52d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.884209 2268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5d883f494049e6a045aacc9f4c52d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:31.886232 2610 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Bootstrap starting.
I20260201 03:08:31.886663 2611 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Bootstrap starting.
I20260201 03:08:31.887089 2610 tablet_bootstrap.cc:654] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.887218 2611 tablet_bootstrap.cc:654] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.887375 2610 log.cc:826] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.887475 2611 log.cc:826] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.887768 2612 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Bootstrap starting.
I20260201 03:08:31.888082 2610 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: No bootstrap required, opened a new log
I20260201 03:08:31.888125 2610 ts_tablet_manager.cc:1403] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:31.888196 2611 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: No bootstrap required, opened a new log
I20260201 03:08:31.888248 2611 ts_tablet_manager.cc:1403] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:31.888358 2612 tablet_bootstrap.cc:654] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:31.888608 2612 log.cc:826] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:31.889437 2612 tablet_bootstrap.cc:492] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: No bootstrap required, opened a new log
I20260201 03:08:31.889488 2612 ts_tablet_manager.cc:1403] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:31.889432 2610 raft_consensus.cc:359] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.889525 2610 raft_consensus.cc:385] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.889544 2610 raft_consensus.cc:740] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e4fc301c1244caa5f91e762b4f5ae7, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.889634 2610 consensus_queue.cc:260] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.889779 2611 raft_consensus.cc:359] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.889917 2611 raft_consensus.cc:385] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.889942 2611 raft_consensus.cc:740] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82027ff8aabf4fbba955813b63c1ed57, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.889936 2610 ts_tablet_manager.cc:1434] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:31.890048 2465 heartbeater.cc:499] Master 127.28.181.62:43235 was elected leader, sending a full tablet report...
I20260201 03:08:31.890048 2611 consensus_queue.cc:260] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.890242 2611 ts_tablet_manager.cc:1434] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:08:31.890271 2596 heartbeater.cc:499] Master 127.28.181.62:43235 was elected leader, sending a full tablet report...
I20260201 03:08:31.890888 2612 raft_consensus.cc:359] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.890975 2612 raft_consensus.cc:385] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:31.891026 2612 raft_consensus.cc:740] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf9a89bf9de424c92f9464339ef3baa, State: Initialized, Role: FOLLOWER
I20260201 03:08:31.891124 2612 consensus_queue.cc:260] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:31.891323 2612 ts_tablet_manager.cc:1434] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:31.891381 2334 heartbeater.cc:499] Master 127.28.181.62:43235 was elected leader, sending a full tablet report...
W20260201 03:08:31.931514 2335 tablet.cc:2378] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:31.999950 2616 raft_consensus.cc:493] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:32.000054 2616 raft_consensus.cc:515] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:32.000397 2616 leader_election.cc:290] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82027ff8aabf4fbba955813b63c1ed57 (127.28.181.3:34713), bbf9a89bf9de424c92f9464339ef3baa (127.28.181.1:35305)
I20260201 03:08:32.001524 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.000599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:57789 (local address 127.28.181.3:34713)
0201 03:08:32.000642 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.000644 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.000654 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:32.000684 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.000685 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.000690 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.000737 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:32.001262 (+ 525us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:32.001264 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:32.001264 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:32.001267 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:32.001301 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:32.001305 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:32.001361 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:32.001452 (+ 91us) server_negotiation.cc:300] Negotiation successful
0201 03:08:32.001468 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260201 03:08:32.001525 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.000668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:46205 (local address 127.28.181.1:35305)
0201 03:08:32.000706 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.000708 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.000805 (+ 97us) server_negotiation.cc:408] Connection header received
0201 03:08:32.000849 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.000850 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.000855 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.000904 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:32.001319 (+ 415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:32.001320 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:32.001321 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:32.001324 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:32.001337 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:32.001341 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:32.001392 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:32.001444 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:08:32.001461 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:32.001524 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.000540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:34713 (local address 127.28.181.2:57789)
0201 03:08:32.000591 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:32.000602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:32.000640 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:32.000755 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:32.000756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:32.000759 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:32.001135 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:32.001136 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:32.001141 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:32.001167 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:32.001385 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:32.001386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:32.001386 (+ 0us) client_negotiation.cc:769] Sending connection context
0201 03:08:32.001433 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:08:32.001449 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260201 03:08:32.001808 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.000562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:35305 (local address 127.28.181.2:46205)
0201 03:08:32.000729 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:32.000736 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:32.000803 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:32.000934 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:32.000934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:32.000936 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:32.001229 (+ 293us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:08:32.001229 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:32.001234 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:08:32.001241 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:32.001394 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:32.001395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:32.001395 (+ 0us) client_negotiation.cc:769] Sending connection context
0201 03:08:32.001437 (+ 42us) client_negotiation.cc:240] Negotiation successful
0201 03:08:32.001448 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":128,"threads_started":1}
I20260201 03:08:32.001861 2550 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d5d883f494049e6a045aacc9f4c52d9" candidate_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82027ff8aabf4fbba955813b63c1ed57" is_pre_election: true
I20260201 03:08:32.002005 2550 raft_consensus.cc:2468] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e4fc301c1244caa5f91e762b4f5ae7 in term 0.
I20260201 03:08:32.002127 2288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d5d883f494049e6a045aacc9f4c52d9" candidate_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf9a89bf9de424c92f9464339ef3baa" is_pre_election: true
I20260201 03:08:32.002166 2353 leader_election.cc:304] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82027ff8aabf4fbba955813b63c1ed57, c1e4fc301c1244caa5f91e762b4f5ae7; no voters:
I20260201 03:08:32.002264 2288 raft_consensus.cc:2468] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e4fc301c1244caa5f91e762b4f5ae7 in term 0.
I20260201 03:08:32.002278 2616 raft_consensus.cc:2804] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:32.002333 2616 raft_consensus.cc:493] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:32.002359 2616 raft_consensus.cc:3060] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:32.002976 2616 raft_consensus.cc:515] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:32.003106 2616 leader_election.cc:290] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 82027ff8aabf4fbba955813b63c1ed57 (127.28.181.3:34713), bbf9a89bf9de424c92f9464339ef3baa (127.28.181.1:35305)
I20260201 03:08:32.003259 2550 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d5d883f494049e6a045aacc9f4c52d9" candidate_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82027ff8aabf4fbba955813b63c1ed57"
I20260201 03:08:32.003289 2288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d5d883f494049e6a045aacc9f4c52d9" candidate_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf9a89bf9de424c92f9464339ef3baa"
I20260201 03:08:32.003326 2550 raft_consensus.cc:3060] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:32.003363 2288 raft_consensus.cc:3060] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:32.004137 2550 raft_consensus.cc:2468] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e4fc301c1244caa5f91e762b4f5ae7 in term 1.
I20260201 03:08:32.004248 2288 raft_consensus.cc:2468] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e4fc301c1244caa5f91e762b4f5ae7 in term 1.
I20260201 03:08:32.004278 2353 leader_election.cc:304] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82027ff8aabf4fbba955813b63c1ed57, c1e4fc301c1244caa5f91e762b4f5ae7; no voters:
I20260201 03:08:32.004420 2616 raft_consensus.cc:2804] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:32.004551 2616 raft_consensus.cc:697] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 1 LEADER]: Becoming Leader. State: Replica: c1e4fc301c1244caa5f91e762b4f5ae7, State: Running, Role: LEADER
I20260201 03:08:32.004638 2616 consensus_queue.cc:237] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } }
I20260201 03:08:32.005501 2146 catalog_manager.cc:5654] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e4fc301c1244caa5f91e762b4f5ae7 (127.28.181.2). New cstate: current_term: 1 leader_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e4fc301c1244caa5f91e762b4f5ae7" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 } health_report { overall_health: UNKNOWN } } }
W20260201 03:08:32.022496 2466 tablet.cc:2378] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:32.030290 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.029402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.28.181.2:33671)
0201 03:08:32.029452 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.029453 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.029461 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:32.029517 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.029518 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.029527 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.029571 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:32.030109 (+ 538us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:32.030110 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:32.030110 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:32.030114 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:32.030126 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:32.030129 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:32.030178 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:32.030220 (+ 42us) server_negotiation.cc:300] Negotiation successful
0201 03:08:32.030233 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:32.031517 2288 raft_consensus.cc:1275] T 6d5d883f494049e6a045aacc9f4c52d9 P bbf9a89bf9de424c92f9464339ef3baa [term 1 FOLLOWER]: Refusing update from remote peer c1e4fc301c1244caa5f91e762b4f5ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:32.031750 2550 raft_consensus.cc:1275] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 1 FOLLOWER]: Refusing update from remote peer c1e4fc301c1244caa5f91e762b4f5ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:32.031783 2616 consensus_queue.cc:1048] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf9a89bf9de424c92f9464339ef3baa" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:32.031989 2616 consensus_queue.cc:1048] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82027ff8aabf4fbba955813b63c1ed57" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:32.033831 2623 mvcc.cc:204] Tried to move back new op lower bound from 7249573118079258624 to 7249573117971832832. Current Snapshot: MvccSnapshot[applied={T|T < 7249573118079258624}]
I20260201 03:08:32.036262 2144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:32.036330 2144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:32.036962 2144 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 860f38a6b0484818b0b491d124ad610a: Sending DeleteTablet for 3 replicas of tablet 6d5d883f494049e6a045aacc9f4c52d9
I20260201 03:08:32.037245 2530 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:32 UTC) from {username='slave'} at 127.0.0.1:60578
I20260201 03:08:32.037268 2399 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:32 UTC) from {username='slave'} at 127.0.0.1:57232
I20260201 03:08:32.037408 2268 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d5d883f494049e6a045aacc9f4c52d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:32 UTC) from {username='slave'} at 127.0.0.1:53418
I20260201 03:08:32.037504 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2206
I20260201 03:08:32.038087 2633 tablet_replica.cc:333] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: stopping tablet replica
I20260201 03:08:32.038189 2633 raft_consensus.cc:2243] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:32.038312 2633 raft_consensus.cc:2272] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:32.038712 2633 ts_tablet_manager.cc:1916] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:32.040258 2633 ts_tablet_manager.cc:1929] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:32.040356 2633 log.cc:1199] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/6d5d883f494049e6a045aacc9f4c52d9
I20260201 03:08:32.040634 2633 ts_tablet_manager.cc:1950] T 6d5d883f494049e6a045aacc9f4c52d9 P 82027ff8aabf4fbba955813b63c1ed57: Deleting consensus metadata
I20260201 03:08:32.041026 2131 catalog_manager.cc:4985] TS 82027ff8aabf4fbba955813b63c1ed57 (127.28.181.3:34713): tablet 6d5d883f494049e6a045aacc9f4c52d9 (table test-table [id=a916231ed91747de9dc2ffb1978313f0]) successfully deleted
I20260201 03:08:32.041914 2634 tablet_replica.cc:333] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: stopping tablet replica
I20260201 03:08:32.041991 2634 raft_consensus.cc:2243] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:32.042083 2634 raft_consensus.cc:2272] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260201 03:08:32.043147 2132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.181.1:35305 (error 108)
I20260201 03:08:32.043263 2634 ts_tablet_manager.cc:1916] T 6d5d883f494049e6a045aacc9f4c52d9 P c1e4fc301c1244caa5f91e762b4f5ae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:32.043466 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2337
I20260201 03:08:32.044008 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.043569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:35305 (local address 127.0.0.1:53420)
0201 03:08:32.043937 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:32.043968 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:35305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260201 03:08:32.044107 2132 catalog_manager.cc:4712] TS bbf9a89bf9de424c92f9464339ef3baa (127.28.181.1:35305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d5d883f494049e6a045aacc9f4c52d9: Network error: Client connection negotiation failed: client connection to 127.28.181.1:35305: connect: Connection refused (error 111)
W20260201 03:08:32.048506 2604 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:32.048382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:33671 (local address 127.0.0.1:57262)
0201 03:08:32.048427 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:32.048435 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:32.048466 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.2:33671: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
I20260201 03:08:32.048712 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2468
I20260201 03:08:32.053120 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2115
2026-02-01T03:08:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (607 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260201 03:08:32.072017 29396 mini_oidc.cc:150] Starting JWKS server
I20260201 03:08:32.072907 29396 webserver.cc:492] Webserver started at https://127.0.0.1:44997/ using document root <none> and password file <none>
I20260201 03:08:32.073007 29396 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:08:32.073172 29396 webserver.cc:492] Webserver started at http://127.0.0.1:41221/ using document root <none> and password file <none>
2026-02-01T03:08:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:32Z Disabled control of system clock
I20260201 03:08:32.079017 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44875
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:44875
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44997/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/testchainca.cert with env {}
W20260201 03:08:32.146143 2642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:32.146275 2642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:32.146293 2642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:32.147500 2642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:32.147536 2642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:32.147547 2642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:32.147558 2642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:32.147568 2642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:08:32.147578 2642 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:08:32.147588 2642 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44997/jwks.json
I20260201 03:08:32.148959 2642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:44875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44997/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:32.149106 2642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:32.149256 2642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:08:32.151423 2642 server_base.cc:1060] running on GCE node
W20260201 03:08:32.151391 2648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:32.151500 2650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:32.151405 2647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:32.151750 2642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:32.151957 2642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:32.153075 2642 hybrid_clock.cc:648] HybridClock initialized: now 1769915312153053 us; error 35 us; skew 500 ppm
I20260201 03:08:32.154223 2642 webserver.cc:492] Webserver started at http://127.28.181.62:45839/ using document root <none> and password file <none>
I20260201 03:08:32.154433 2642 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:32.154471 2642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:32.154589 2642 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:32.155465 2642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "1af38e1c2f0e4714ab568a25abc4f7f6"
format_stamp: "Formatted at 2026-02-01 03:08:32 on dist-test-slave-ck1n"
server_key: "650ba79b898450ecf438e29a3d6f6327"
server_key_iv: "55dd02ebff6401bfb9ddf40e6318eebe"
server_key_version: "encryptionkey@0"
I20260201 03:08:32.155794 2642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "1af38e1c2f0e4714ab568a25abc4f7f6"
format_stamp: "Formatted at 2026-02-01 03:08:32 on dist-test-slave-ck1n"
server_key: "650ba79b898450ecf438e29a3d6f6327"
server_key_iv: "55dd02ebff6401bfb9ddf40e6318eebe"
server_key_version: "encryptionkey@0"
I20260201 03:08:32.156975 2642 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260201 03:08:32.157570 2656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:32.157733 2642 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:32.157843 2642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "1af38e1c2f0e4714ab568a25abc4f7f6"
format_stamp: "Formatted at 2026-02-01 03:08:32 on dist-test-slave-ck1n"
server_key: "650ba79b898450ecf438e29a3d6f6327"
server_key_iv: "55dd02ebff6401bfb9ddf40e6318eebe"
server_key_version: "encryptionkey@0"
I20260201 03:08:32.157914 2642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:32.176155 2642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:32.176379 2642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:32.176489 2642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:32.179821 2642 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:44875
I20260201 03:08:32.179845 2708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:44875 every 8 connection(s)
I20260201 03:08:32.180099 2642 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:32.180572 2709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:32.182058 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2642
I20260201 03:08:32.182158 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:32.182312 29396 external_mini_cluster.cc:1468] Setting key 4f218db1a3ae7ac6de12c8b01745490d
I20260201 03:08:32.182850 2709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6: Bootstrap starting.
I20260201 03:08:32.183415 2709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:32.183661 2709 log.cc:826] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:32.184231 2709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6: No bootstrap required, opened a new log
I20260201 03:08:32.185081 2709 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } }
I20260201 03:08:32.185223 2709 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:32.185252 2709 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af38e1c2f0e4714ab568a25abc4f7f6, State: Initialized, Role: FOLLOWER
I20260201 03:08:32.185361 2709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } }
I20260201 03:08:32.185433 2709 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:32.185479 2709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:32.185530 2709 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:32.186101 2709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } }
I20260201 03:08:32.186235 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.182719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.28.181.62:44875)
0201 03:08:32.182903 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.182909 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.182928 (+ 19us) server_negotiation.cc:408] Connection header received
0201 03:08:32.183296 (+ 368us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.183302 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.183354 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.183477 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:32.183817 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.184462 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:32.184986 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.185201 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:32.185616 (+ 415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:32.185623 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:32.185623 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:32.185825 (+ 202us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:32.185853 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:32.185860 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:32.185938 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:32.186004 (+ 66us) server_negotiation.cc:300] Negotiation successful
0201 03:08:32.186052 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260201 03:08:32.186221 2709 leader_election.cc:304] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af38e1c2f0e4714ab568a25abc4f7f6; no voters:
I20260201 03:08:32.186375 2709 leader_election.cc:290] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:32.186415 2714 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:32.186530 2714 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af38e1c2f0e4714ab568a25abc4f7f6, State: Running, Role: LEADER
I20260201 03:08:32.186614 2714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } }
I20260201 03:08:32.186812 2709 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:32.186889 2715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af38e1c2f0e4714ab568a25abc4f7f6. Latest consensus state: current_term: 1 leader_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } } }
I20260201 03:08:32.186951 2715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:32.187004 29396 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:08:32.187085 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af38e1c2f0e4714ab568a25abc4f7f6" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 44875 } } }
I20260201 03:08:32.187129 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:32.187193 2718 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:32.187608 2718 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:32.189081 2718 catalog_manager.cc:1357] Generated new cluster ID: ca9eab12b36c4167bee13f72ee2805f8
I20260201 03:08:32.189144 2718 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:32.196171 2718 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:32.196797 2718 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:32.206037 2718 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1af38e1c2f0e4714ab568a25abc4f7f6: Generated new TSK 0
I20260201 03:08:32.206174 2718 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:08:32.233878 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.212091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.28.181.62:44875)
0201 03:08:32.212132 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.212133 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.212210 (+ 77us) server_negotiation.cc:408] Connection header received
0201 03:08:32.212294 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.212295 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.212305 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.212353 (+ 48us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:32.212576 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.212974 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:32.213516 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.213649 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:32.213676 (+ 27us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:32.233559 (+ 19883us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0201 03:08:32.233649 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:32.233748 (+ 99us) server_negotiation.cc:300] Negotiation successful
0201 03:08:32.233789 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21,"thread_start_us":112,"threads_started":1}
I20260201 03:08:32.239575 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:32.237821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.28.181.62:44875)
0201 03:08:32.237856 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:32.237858 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:32.237883 (+ 25us) server_negotiation.cc:408] Connection header received
0201 03:08:32.237946 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:32.237947 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:32.237954 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:32.237996 (+ 42us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:32.238204 (+ 208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.238577 (+ 373us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:32.239060 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.239179 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:32.239194 (+ 15us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:32.239414 (+ 220us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0201 03:08:32.239521 (+ 107us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36422: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":17}
W20260201 03:08:32.239622 2712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36422: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260201 03:08:32.239732 2745 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:32.237732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44875 (local address 127.0.0.1:36422)
0201 03:08:32.237832 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:32.237842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:32.237896 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:32.237998 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:32.237998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:32.238000 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:08:32.238141 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:32.238142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.238656 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:32.238657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:32.238982 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:32.238984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:32.239054 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:32.239057 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:08:32.239507 (+ 450us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0201 03:08:32.239570 (+ 63us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.181.62:44875: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":68,"threads_started":1}
I20260201 03:08:47.244073 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.242382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.28.181.62:44875)
0201 03:08:47.242502 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.242504 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.242520 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:08:47.242560 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.242561 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.242570 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.242614 (+ 44us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:47.242830 (+ 216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.243256 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.243708 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.243798 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.243812 (+ 14us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:47.243951 (+ 139us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0201 03:08:47.244031 (+ 80us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54472: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260201 03:08:47.244122 2751 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54472: JWT verification failed: token expired
W20260201 03:08:47.244149 2750 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.242283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:44875 (local address 127.0.0.1:54472)
0201 03:08:47.242421 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:47.242431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:47.242512 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:47.242631 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:47.242632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:47.242633 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:08:47.242774 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.242776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.243313 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.243313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:47.243644 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.243646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.243700 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.243703 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:08:47.244009 (+ 306us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0201 03:08:47.244075 (+ 66us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.181.62:44875: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
W20260201 03:08:47.245326 2751 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.245043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.28.181.62:44875)
0201 03:08:47.245089 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.245091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.245143 (+ 52us) server_negotiation.cc:408] Connection header received
0201 03:08:47.245180 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.245180 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.245186 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.245231 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:47.245298 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54488: BlockingRecv error: recv got EOF from 127.0.0.1:54488 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260201 03:08:47.247750 2761 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.28.181.62:44875
W20260201 03:08:47.248051 2751 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.247614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.28.181.62:44875)
0201 03:08:47.247646 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.247648 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.247732 (+ 84us) server_negotiation.cc:408] Connection header received
0201 03:08:47.247856 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.247857 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.247864 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.247928 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:47.248015 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54496: BlockingRecv error: recv got EOF from 127.0.0.1:54496 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260201 03:08:47.248143 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2642
2026-02-01T03:08:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260201 03:08:47.285549 29396 mini_oidc.cc:150] Starting JWKS server
I20260201 03:08:47.286197 29396 webserver.cc:492] Webserver started at https://127.0.0.1:43491/ using document root <none> and password file <none>
I20260201 03:08:47.286233 29396 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:08:47.286350 29396 webserver.cc:492] Webserver started at http://127.0.0.1:39461/ using document root <none> and password file <none>
2026-02-01T03:08:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:47Z Disabled control of system clock
I20260201 03:08:47.291990 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38549
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:38549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/testchainca.cert with env {}
W20260201 03:08:47.356830 2767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:47.356947 2767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:47.356966 2767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:47.358189 2767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:47.358224 2767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:47.358235 2767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:47.358268 2767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:47.358294 2767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:08:47.358306 2767 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:08:47.358327 2767 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43491/jwks.json
I20260201 03:08:47.359663 2767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:38549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:47.359789 2767 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:47.359934 2767 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:47.362120 2772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:47.362172 2767 server_base.cc:1060] running on GCE node
W20260201 03:08:47.362128 2773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:47.362236 2775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:47.362445 2767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:47.362656 2767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:47.363883 2767 hybrid_clock.cc:648] HybridClock initialized: now 1769915327363870 us; error 27 us; skew 500 ppm
I20260201 03:08:47.364828 2767 webserver.cc:492] Webserver started at http://127.28.181.62:40341/ using document root <none> and password file <none>
I20260201 03:08:47.365000 2767 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:47.365054 2767 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:47.365159 2767 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:47.365928 2767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "16555f6f322e466d862257a187093340"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "5780fb2e164259f205076e38493b5021"
server_key_iv: "e4731b61881bac6ab4ec150551d82a70"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.366197 2767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "16555f6f322e466d862257a187093340"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "5780fb2e164259f205076e38493b5021"
server_key_iv: "e4731b61881bac6ab4ec150551d82a70"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.367200 2767 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:47.367805 2781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:47.367966 2767 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:47.368029 2767 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "16555f6f322e466d862257a187093340"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "5780fb2e164259f205076e38493b5021"
server_key_iv: "e4731b61881bac6ab4ec150551d82a70"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.368068 2767 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:47.387046 2767 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:47.387195 2767 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:47.387262 2767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:47.390159 2767 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:38549
I20260201 03:08:47.390206 2833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:38549 every 8 connection(s)
I20260201 03:08:47.390450 2767 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:47.390951 2834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:47.392709 2834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: Bootstrap starting.
I20260201 03:08:47.393203 2834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:47.393417 2834 log.cc:826] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:47.394078 2834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: No bootstrap required, opened a new log
I20260201 03:08:47.394902 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2767
I20260201 03:08:47.394968 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:47.395109 29396 external_mini_cluster.cc:1468] Setting key 7daad1043c6873d82f2d441263117a0b
I20260201 03:08:47.395157 2834 raft_consensus.cc:359] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } }
I20260201 03:08:47.395303 2834 raft_consensus.cc:385] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:47.395337 2834 raft_consensus.cc:740] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16555f6f322e466d862257a187093340, State: Initialized, Role: FOLLOWER
I20260201 03:08:47.395443 2834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } }
I20260201 03:08:47.395514 2834 raft_consensus.cc:399] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:47.395550 2834 raft_consensus.cc:493] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:47.395592 2834 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:47.396067 2834 raft_consensus.cc:515] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } }
I20260201 03:08:47.396166 2834 leader_election.cc:304] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16555f6f322e466d862257a187093340; no voters:
I20260201 03:08:47.396304 2834 leader_election.cc:290] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:47.396334 2839 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:47.396462 2839 raft_consensus.cc:697] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [term 1 LEADER]: Becoming Leader. State: Replica: 16555f6f322e466d862257a187093340, State: Running, Role: LEADER
I20260201 03:08:47.396504 2834 sys_catalog.cc:565] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:47.396533 2839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } }
I20260201 03:08:47.396790 2839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16555f6f322e466d862257a187093340. Latest consensus state: current_term: 1 leader_uuid: "16555f6f322e466d862257a187093340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } } }
I20260201 03:08:47.396881 2839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:47.396798 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16555f6f322e466d862257a187093340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16555f6f322e466d862257a187093340" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38549 } } }
I20260201 03:08:47.397007 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340 [sys.catalog]: This master's current role is: LEADER
W20260201 03:08:47.398304 2854 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:47.398361 2854 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:47.398690 2844 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:47.398815 2844 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:47.399755 2844 catalog_manager.cc:1357] Generated new cluster ID: 6de67b21c6cf4d03ae6434697fb08d4d
I20260201 03:08:47.399803 2844 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:47.402926 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.395605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.28.181.62:38549)
0201 03:08:47.395756 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.395761 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.395776 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:08:47.395842 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.395845 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.395879 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.395994 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:47.396387 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.397354 (+ 967us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.397956 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.398261 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.402400 (+ 4139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:47.402426 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:47.402426 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:47.402586 (+ 160us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:47.402607 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:47.402612 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:47.402695 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:47.402770 (+ 75us) server_negotiation.cc:300] Negotiation successful
0201 03:08:47.402819 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260201 03:08:47.403621 29396 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:08:47.407864 2844 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:47.408423 2844 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:47.412611 2844 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 16555f6f322e466d862257a187093340: Generated new TSK 0
I20260201 03:08:47.412735 2844 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260201 03:08:47.431743 2864 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260201 03:08:47.431809 2864 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260201 03:08:47.431993 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.419805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.28.181.62:38549)
0201 03:08:47.419838 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.419840 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.419935 (+ 95us) server_negotiation.cc:408] Connection header received
0201 03:08:47.419990 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.419991 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.419999 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.420042 (+ 43us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:47.420314 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.420706 (+ 392us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.421376 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.421467 (+ 91us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.421491 (+ 24us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:47.431851 (+ 10360us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:08:47.431945 (+ 94us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260201 03:08:47.432041 2838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260201 03:08:47.432286 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.419720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38549 (local address 127.0.0.1:56298)
0201 03:08:47.419868 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:47.419877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:47.419937 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:47.420060 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:47.420061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:47.420062 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:08:47.420256 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.420258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.420767 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.420768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:47.421306 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.421308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.421366 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.421370 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:08:47.432120 (+ 10750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:08:47.432194 (+ 74us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.181.62:38549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260201 03:08:47.432518 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2767
2026-02-01T03:08:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (183 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260201 03:08:47.468691 29396 mini_oidc.cc:150] Starting JWKS server
I20260201 03:08:47.469267 29396 webserver.cc:492] Webserver started at https://127.0.0.1:38719/ using document root <none> and password file <none>
I20260201 03:08:47.469300 29396 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:08:47.469410 29396 webserver.cc:492] Webserver started at http://127.0.0.1:45263/ using document root <none> and password file <none>
2026-02-01T03:08:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:47Z Disabled control of system clock
I20260201 03:08:47.474929 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:37485
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:37485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38719/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260201 03:08:47.540119 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:47.540258 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:47.540287 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:47.541452 2870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:47.541507 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:47.541527 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:47.541553 2870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:47.541574 2870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:08:47.541599 2870 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:08:47.541620 2870 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38719/jwks.json
I20260201 03:08:47.542917 2870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:37485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:37485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38719/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:47.543107 2870 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:47.543290 2870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:47.545161 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:47.545179 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:47.545239 2870 server_base.cc:1060] running on GCE node
W20260201 03:08:47.545171 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:47.545469 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:47.545663 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:47.546779 2870 hybrid_clock.cc:648] HybridClock initialized: now 1769915327546761 us; error 33 us; skew 500 ppm
I20260201 03:08:47.547669 2870 webserver.cc:492] Webserver started at http://127.28.181.62:35521/ using document root <none> and password file <none>
I20260201 03:08:47.547832 2870 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:47.547868 2870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:47.547967 2870 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:47.548682 2870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "0a1b22aa565c4ce6afddc0e755087183"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "527ce3be473e75212d71d11ac5529fc4"
server_key_iv: "97d1a05b274ede649e2a9ab8b5430427"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.548942 2870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "0a1b22aa565c4ce6afddc0e755087183"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "527ce3be473e75212d71d11ac5529fc4"
server_key_iv: "97d1a05b274ede649e2a9ab8b5430427"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.549932 2870 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:47.550509 2884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:47.550664 2870 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:47.550717 2870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "0a1b22aa565c4ce6afddc0e755087183"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "527ce3be473e75212d71d11ac5529fc4"
server_key_iv: "97d1a05b274ede649e2a9ab8b5430427"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.550786 2870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:47.573146 2870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:47.573330 2870 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:47.573433 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:47.576491 2870 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:37485
I20260201 03:08:47.576520 2936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:37485 every 8 connection(s)
I20260201 03:08:47.576776 2870 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:47.577235 2937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:47.577590 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2870
I20260201 03:08:47.577672 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:47.577841 29396 external_mini_cluster.cc:1468] Setting key 7856c9946d145f0b075bfb30ef78b5ee
I20260201 03:08:47.579391 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183: Bootstrap starting.
I20260201 03:08:47.579840 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:47.580067 2937 log.cc:826] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:47.580636 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183: No bootstrap required, opened a new log
I20260201 03:08:47.581336 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.578246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.28.181.62:37485)
0201 03:08:47.578429 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.578437 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.578455 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:08:47.578703 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.578707 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.578743 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.578862 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:47.579186 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.579810 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.580345 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.580555 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.580887 (+ 332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:47.580893 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:47.580893 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:47.581024 (+ 131us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:47.581046 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:47.581051 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:47.581130 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:47.581189 (+ 59us) server_negotiation.cc:300] Negotiation successful
0201 03:08:47.581230 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260201 03:08:47.581751 2937 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } }
I20260201 03:08:47.581889 2937 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:47.581923 2937 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1b22aa565c4ce6afddc0e755087183, State: Initialized, Role: FOLLOWER
I20260201 03:08:47.582000 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } }
I20260201 03:08:47.582075 2937 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:47.582108 2937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:47.582129 29396 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:08:47.582142 2937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:47.582674 2937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } }
I20260201 03:08:47.582764 2937 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a1b22aa565c4ce6afddc0e755087183; no voters:
I20260201 03:08:47.582903 2937 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:47.582931 2942 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:47.583098 2942 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [term 1 LEADER]: Becoming Leader. State: Replica: 0a1b22aa565c4ce6afddc0e755087183, State: Running, Role: LEADER
I20260201 03:08:47.583110 2937 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:47.583200 2942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } }
I20260201 03:08:47.583483 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a1b22aa565c4ce6afddc0e755087183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } } }
I20260201 03:08:47.583500 2944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1b22aa565c4ce6afddc0e755087183. Latest consensus state: current_term: 1 leader_uuid: "0a1b22aa565c4ce6afddc0e755087183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1b22aa565c4ce6afddc0e755087183" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 37485 } } }
I20260201 03:08:47.583561 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:47.583588 2944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:47.583882 2948 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:47.584264 2948 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:47.585422 2948 catalog_manager.cc:1357] Generated new cluster ID: b6c41c7cab1a422983e7a6a06b604c5f
I20260201 03:08:47.585467 2948 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:47.593451 2948 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:47.594036 2948 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:47.598744 2948 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0a1b22aa565c4ce6afddc0e755087183: Generated new TSK 0
I20260201 03:08:47.598877 2948 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260201 03:08:47.622624 2967 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260201 03:08:47.622674 2967 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260201 03:08:47.623224 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.603898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.28.181.62:37485)
0201 03:08:47.603932 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.603933 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.604036 (+ 103us) server_negotiation.cc:408] Connection header received
0201 03:08:47.604091 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.604091 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.604108 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.604152 (+ 44us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:47.604420 (+ 268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.604802 (+ 382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.605478 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.605571 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.605597 (+ 26us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:47.623090 (+ 17493us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:08:47.623191 (+ 101us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":19}
W20260201 03:08:47.623276 2940 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260201 03:08:47.623315 2966 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.603817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:37485 (local address 127.0.0.1:47668)
0201 03:08:47.603966 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:47.603975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:47.604034 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:47.604168 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:47.604169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:47.604171 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:08:47.604363 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.604364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.604875 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.604876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:47.605407 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.605409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.605471 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.605475 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:08:47.623151 (+ 17676us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:08:47.623230 (+ 79us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.181.62:37485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260201 03:08:47.623605 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2870
2026-02-01T03:08:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (191 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-02-01T03:08:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:47Z Disabled control of system clock
I20260201 03:08:47.665431 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:42899
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:42899
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/testchainca.cert with env {}
W20260201 03:08:47.729575 2971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:47.729712 2971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:47.729741 2971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:47.730921 2971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:47.730975 2971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:47.730995 2971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:47.731024 2971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:47.731050 2971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:08:47.731074 2971 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:08:47.731096 2971 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260201 03:08:47.732384 2971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:42899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:47.732568 2971 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:47.732707 2971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:47.734405 2976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:47.734417 2977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:47.734422 2979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:47.734575 2971 server_base.cc:1060] running on GCE node
I20260201 03:08:47.734741 2971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:47.734972 2971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:47.736109 2971 hybrid_clock.cc:648] HybridClock initialized: now 1769915327736099 us; error 33 us; skew 500 ppm
I20260201 03:08:47.737020 2971 webserver.cc:492] Webserver started at http://127.28.181.62:45553/ using document root <none> and password file <none>
I20260201 03:08:47.737190 2971 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:47.737243 2971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:47.737355 2971 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:47.738178 2971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "270a429f31f044c2b1a76c21c02d062c"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "b68325c71066348f24078ddbae2587a3"
server_key_iv: "379b432a943224882e4e7584e79024f2"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.738447 2971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "270a429f31f044c2b1a76c21c02d062c"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "b68325c71066348f24078ddbae2587a3"
server_key_iv: "379b432a943224882e4e7584e79024f2"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.739391 2971 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:47.739943 2985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:47.740082 2971 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:47.740154 2971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "270a429f31f044c2b1a76c21c02d062c"
format_stamp: "Formatted at 2026-02-01 03:08:47 on dist-test-slave-ck1n"
server_key: "b68325c71066348f24078ddbae2587a3"
server_key_iv: "379b432a943224882e4e7584e79024f2"
server_key_version: "encryptionkey@0"
I20260201 03:08:47.740217 2971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:47.747835 2971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:47.748037 2971 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:47.748142 2971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:47.751106 2971 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:42899
I20260201 03:08:47.751127 3037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:42899 every 8 connection(s)
I20260201 03:08:47.751365 2971 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:47.751837 3038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:47.753525 3038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c: Bootstrap starting.
I20260201 03:08:47.753995 3038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:47.754166 3038 log.cc:826] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:47.754652 3038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c: No bootstrap required, opened a new log
I20260201 03:08:47.755551 3038 raft_consensus.cc:359] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } }
I20260201 03:08:47.755640 3038 raft_consensus.cc:385] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:47.755656 3038 raft_consensus.cc:740] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270a429f31f044c2b1a76c21c02d062c, State: Initialized, Role: FOLLOWER
I20260201 03:08:47.755744 3038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } }
I20260201 03:08:47.755803 3038 raft_consensus.cc:399] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:47.755822 3038 raft_consensus.cc:493] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:47.755843 3038 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:47.756251 3038 raft_consensus.cc:515] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } }
I20260201 03:08:47.756320 3038 leader_election.cc:304] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 270a429f31f044c2b1a76c21c02d062c; no voters:
I20260201 03:08:47.756428 3038 leader_election.cc:290] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:47.756474 3041 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:47.756584 3041 raft_consensus.cc:697] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [term 1 LEADER]: Becoming Leader. State: Replica: 270a429f31f044c2b1a76c21c02d062c, State: Running, Role: LEADER
I20260201 03:08:47.756630 3038 sys_catalog.cc:565] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:47.756649 3041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } }
I20260201 03:08:47.756906 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "270a429f31f044c2b1a76c21c02d062c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } } }
I20260201 03:08:47.756906 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 270a429f31f044c2b1a76c21c02d062c. Latest consensus state: current_term: 1 leader_uuid: "270a429f31f044c2b1a76c21c02d062c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270a429f31f044c2b1a76c21c02d062c" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42899 } } }
I20260201 03:08:47.756990 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:47.757016 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:47.757155 3046 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:47.757561 3046 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:47.758208 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 2971
I20260201 03:08:47.758296 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:47.758438 29396 external_mini_cluster.cc:1468] Setting key 9ca90fed3a4c1ea50e2da7f1840fad89
I20260201 03:08:47.759161 3046 catalog_manager.cc:1357] Generated new cluster ID: ff0a9a7a339e4c709d2e5ff184893dc5
I20260201 03:08:47.759214 3046 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:47.762387 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.758789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.28.181.62:42899)
0201 03:08:47.758978 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.758984 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.759005 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:08:47.759086 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.759090 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.759153 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.759252 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:47.759703 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.760663 (+ 960us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.761285 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.761746 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.761866 (+ 120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:47.761887 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:47.761888 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:08:47.762067 (+ 179us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:47.762089 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:08:47.762096 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:47.762161 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:47.762226 (+ 65us) server_negotiation.cc:300] Negotiation successful
0201 03:08:47.762270 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260201 03:08:47.763125 29396 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:08:47.769862 3046 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:47.770260 3046 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:47.782030 3046 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 270a429f31f044c2b1a76c21c02d062c: Generated new TSK 0
I20260201 03:08:47.782159 3046 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:08:47.872619 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:47.788131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.28.181.62:42899)
0201 03:08:47.788163 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:47.788165 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:47.788251 (+ 86us) server_negotiation.cc:408] Connection header received
0201 03:08:47.788280 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:47.788281 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:47.788291 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:47.788341 (+ 50us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:08:47.788529 (+ 188us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.789391 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.789872 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.790022 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.790056 (+ 34us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:08:47.872403 (+ 82347us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0201 03:08:47.872554 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260201 03:08:47.872685 3067 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:47.788057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42899 (local address 127.0.0.1:45884)
0201 03:08:47.788170 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:47.788178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:47.788242 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:47.788355 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:47.788356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:47.788357 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:08:47.788476 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.788477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.789453 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:47.789454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:47.789802 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:47.789805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:47.789860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:47.789863 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:08:47.872499 (+ 82636us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0201 03:08:47.872599 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.181.62:42899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":77,"threads_started":1}
W20260201 03:08:47.872813 3061 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20260201 03:08:47.872956 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 2971
2026-02-01T03:08:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (229 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (71 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (69 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:48 dist-test-slave-ck1n krb5kdc[3079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:48 dist-test-slave-ck1n krb5kdc[3079](info): set up 2 sockets
Feb 01 03:08:48 dist-test-slave-ck1n krb5kdc[3079](info): commencing operation
krb5kdc: starting...
W20260201 03:08:49.729940 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.699s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:49 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:49.833067 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:32887
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:49.897434 3095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:49.897558 3095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:49.897576 3095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:49.898715 3095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:49.898748 3095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:49.898761 3095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:49.898770 3095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:49.898780 3095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:49.900044 3095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:32887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:49.900175 3095 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:49.900276 3095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:49.902006 3101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:49.902066 3100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:49.902014 3103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:49.902148 3095 server_base.cc:1060] running on GCE node
I20260201 03:08:49.902316 3095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:49.902511 3095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:49.903626 3095 hybrid_clock.cc:648] HybridClock initialized: now 1769915329903612 us; error 28 us; skew 500 ppm
Feb 01 03:08:49 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:49.905397 3095 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:08:49.905890 3095 webserver.cc:492] Webserver started at http://127.28.181.62:36891/ using document root <none> and password file <none>
I20260201 03:08:49.906061 3095 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:49.906091 3095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:49.906191 3095 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:49.906909 3095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "2a7987feea9a41a7a6eb099938d83bda"
format_stamp: "Formatted at 2026-02-01 03:08:49 on dist-test-slave-ck1n"
server_key: "d6b1396aaa7d0b4dd293232d941196dc"
server_key_iv: "f43c13a50bc1173d23b32d8be36f109b"
server_key_version: "encryptionkey@0"
I20260201 03:08:49.907171 3095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "2a7987feea9a41a7a6eb099938d83bda"
format_stamp: "Formatted at 2026-02-01 03:08:49 on dist-test-slave-ck1n"
server_key: "d6b1396aaa7d0b4dd293232d941196dc"
server_key_iv: "f43c13a50bc1173d23b32d8be36f109b"
server_key_version: "encryptionkey@0"
I20260201 03:08:49.908167 3095 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:49.908694 3110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:49.908857 3095 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:49.908910 3095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "2a7987feea9a41a7a6eb099938d83bda"
format_stamp: "Formatted at 2026-02-01 03:08:49 on dist-test-slave-ck1n"
server_key: "d6b1396aaa7d0b4dd293232d941196dc"
server_key_iv: "f43c13a50bc1173d23b32d8be36f109b"
server_key_version: "encryptionkey@0"
I20260201 03:08:49.908982 3095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:49.920253 3095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:49.921499 3095 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:49.921609 3095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:49.924656 3095 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:32887
I20260201 03:08:49.924679 3162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:32887 every 8 connection(s)
I20260201 03:08:49.924909 3095 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:49.925364 3163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:49.925930 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3095
I20260201 03:08:49.926026 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:49.926195 29396 external_mini_cluster.cc:1468] Setting key fc9b134080572167f8b90907be3bbcf6
I20260201 03:08:49.928078 3163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda: Bootstrap starting.
I20260201 03:08:49.928532 3163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:49.928764 3163 log.cc:826] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:49.929342 3163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda: No bootstrap required, opened a new log
I20260201 03:08:49.930684 3163 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } }
I20260201 03:08:49.930771 3163 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:49.930794 3163 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7987feea9a41a7a6eb099938d83bda, State: Initialized, Role: FOLLOWER
I20260201 03:08:49.930882 3163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } }
I20260201 03:08:49.930925 3163 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:49.930943 3163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:49.930996 3163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 0 FOLLOWER]: Advancing to term 1
Feb 01 03:08:49 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:49.931502 3163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } }
I20260201 03:08:49.931602 3163 leader_election.cc:304] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a7987feea9a41a7a6eb099938d83bda; no voters:
I20260201 03:08:49.931742 3163 leader_election.cc:290] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:49.931769 3168 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:49.931902 3168 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [term 1 LEADER]: Becoming Leader. State: Replica: 2a7987feea9a41a7a6eb099938d83bda, State: Running, Role: LEADER
I20260201 03:08:49.931978 3163 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:49.931975 3168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } }
I20260201 03:08:49.932302 3169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a7987feea9a41a7a6eb099938d83bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } } }
I20260201 03:08:49.932346 3169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:49.932466 3170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7987feea9a41a7a6eb099938d83bda. Latest consensus state: current_term: 1 leader_uuid: "2a7987feea9a41a7a6eb099938d83bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7987feea9a41a7a6eb099938d83bda" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 32887 } } }
I20260201 03:08:49.932495 3172 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:49.932504 3170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:49.932868 3172 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:49.934041 3172 catalog_manager.cc:1357] Generated new cluster ID: 62ef678c61824f8790107d615898fa62
I20260201 03:08:49.934079 3172 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:49.934545 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:49.926643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.28.181.62:32887)
0201 03:08:49.926805 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:49.926812 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:49.926829 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:08:49.927129 (+ 300us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:49.927134 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:49.927165 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:49.927255 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:49.928095 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:49.928715 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:49.929338 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:49.929577 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:49.931489 (+ 1912us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:49.931496 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:49.931496 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:49.931513 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:49.933283 (+ 1770us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:49.933612 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:49.933615 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:49.933615 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:49.933667 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:49.933847 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:49.933848 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:49.933848 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:49.934045 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:49.934188 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:49.934288 (+ 100us) server_negotiation.cc:300] Negotiation successful
0201 03:08:49.934386 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260201 03:08:49.945389 3172 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:49.945842 3172 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:49.951256 3172 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2a7987feea9a41a7a6eb099938d83bda: Generated new TSK 0
I20260201 03:08:49.951395 3172 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:49.992991 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:32887
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:50.072474 3191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:50.072619 3191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:50.072649 3191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:50.073875 3191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:50.073926 3191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:50.073963 3191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:50.075305 3191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:50.075512 3191 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:50.075666 3191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:50.077733 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:50.077723 3199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:50.077747 3196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:50.078506 3191 server_base.cc:1060] running on GCE node
I20260201 03:08:50.078685 3191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:50.078909 3191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:50.080039 3191 hybrid_clock.cc:648] HybridClock initialized: now 1769915330080022 us; error 34 us; skew 500 ppm
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.082015 3191 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:08:50.082487 3191 webserver.cc:492] Webserver started at http://127.28.181.1:37589/ using document root <none> and password file <none>
I20260201 03:08:50.082628 3191 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:50.082655 3191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:50.082719 3191 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:50.083415 3191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "402b101e2bc146a392ab81194397824b"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "5573ad7a75e6259d1c2b368b89dbe88f"
server_key_iv: "650f50b001eef031d27c084907b6ef4a"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.083631 3191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "402b101e2bc146a392ab81194397824b"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "5573ad7a75e6259d1c2b368b89dbe88f"
server_key_iv: "650f50b001eef031d27c084907b6ef4a"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.084546 3191 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:50.085088 3206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.085218 3191 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:50.085275 3191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "402b101e2bc146a392ab81194397824b"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "5573ad7a75e6259d1c2b368b89dbe88f"
server_key_iv: "650f50b001eef031d27c084907b6ef4a"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.085315 3191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:50.094868 3191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:50.096088 3191 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:50.096168 3191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:50.096349 3191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:50.096546 3191 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:50.096570 3191 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.096586 3191 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:50.096596 3191 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.101367 3191 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:41795
I20260201 03:08:50.101397 3319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:41795 every 8 connection(s)
I20260201 03:08:50.101658 3191 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:50.106431 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3191
I20260201 03:08:50.106523 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:50.106673 29396 external_mini_cluster.cc:1468] Setting key 7f5987505fcc0fb736011ca1a3f1c2a5
I20260201 03:08:50.109191 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.102356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:35293 (local address 127.28.181.62:32887)
0201 03:08:50.102434 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.102436 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.102561 (+ 125us) server_negotiation.cc:408] Connection header received
0201 03:08:50.103016 (+ 455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.103017 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.103027 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.103077 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.103685 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.104146 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.104647 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.104797 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.106740 (+ 1943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.106743 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.106744 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.106751 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.108163 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.108490 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.108491 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.108491 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.108516 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.108709 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.108710 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.108710 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.108774 (+ 64us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.108822 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.109081 (+ 259us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.109139 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:08:50.109287 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.102318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:32887 (local address 127.28.181.1:35293)
0201 03:08:50.102466 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:50.102489 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:50.102934 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:50.103129 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:50.103131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:50.103354 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:50.103596 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.103598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.104227 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.104228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:50.104575 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.104577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.104697 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.105428 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:50.105441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:50.106663 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:50.108234 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.108239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.108240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.108396 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.108582 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.108583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.108583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.108651 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.108833 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:50.108837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:50.108966 (+ 129us) client_negotiation.cc:769] Sending connection context
0201 03:08:50.109062 (+ 96us) client_negotiation.cc:240] Negotiation successful
0201 03:08:50.109149 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260201 03:08:50.109675 3320 heartbeater.cc:344] Connected to a master server at 127.28.181.62:32887
I20260201 03:08:50.109752 3320 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:50.109939 3320 heartbeater.cc:507] Master 127.28.181.62:32887 requested a full tablet report, sending...
I20260201 03:08:50.110329 3127 ts_manager.cc:194] Registered new tserver with Master: 402b101e2bc146a392ab81194397824b (127.28.181.1:41795)
I20260201 03:08:50.110863 3127 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:35293
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:50.141628 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:32887
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:50.206501 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:50.206629 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:50.206646 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:50.207872 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:50.207907 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:50.207930 3327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:50.209187 3327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:50.209324 3327 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:50.209427 3327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:50.211390 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:50.211395 3327 server_base.cc:1060] running on GCE node
W20260201 03:08:50.211474 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:50.211423 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:50.211658 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:50.211838 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:50.212975 3327 hybrid_clock.cc:648] HybridClock initialized: now 1769915330212928 us; error 64 us; skew 500 ppm
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.214938 3327 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:08:50.215359 3327 webserver.cc:492] Webserver started at http://127.28.181.2:40577/ using document root <none> and password file <none>
I20260201 03:08:50.215523 3327 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:50.215570 3327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:50.215674 3327 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:50.216428 3327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "b289cc29abbe4431827d063c9b6930ac"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "152eb4d620a7a6124518946d564727b0"
server_key_iv: "0e4ace0a537e30375dfe155a5b4dd617"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.216697 3327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "b289cc29abbe4431827d063c9b6930ac"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "152eb4d620a7a6124518946d564727b0"
server_key_iv: "0e4ace0a537e30375dfe155a5b4dd617"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.217728 3327 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:50.218357 3342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.218508 3327 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.218581 3327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "b289cc29abbe4431827d063c9b6930ac"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "152eb4d620a7a6124518946d564727b0"
server_key_iv: "0e4ace0a537e30375dfe155a5b4dd617"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.218643 3327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:50.226086 3327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:50.227252 3327 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:50.227370 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:50.227540 3327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:50.227775 3327 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:50.227823 3327 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.227857 3327 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:50.227878 3327 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.232362 3327 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:33523
I20260201 03:08:50.232415 3455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:33523 every 8 connection(s)
I20260201 03:08:50.232643 3327 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:50.234498 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3327
I20260201 03:08:50.234565 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:50.234694 29396 external_mini_cluster.cc:1468] Setting key 3f049efc0a8d8c386f32be477c6d0d9a
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:50.240032 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.233184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:32887 (local address 127.28.181.2:41909)
0201 03:08:50.233287 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:50.233325 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:50.233658 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:50.233858 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:50.233861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:50.234076 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:50.234292 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.234294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.234972 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.234974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:50.235406 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.235408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.235522 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.236279 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:50.236298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:50.237562 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:50.239140 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.239144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.239145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.239297 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.239450 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.239450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.239451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.239492 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.239710 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:50.239712 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:50.239788 (+ 76us) client_negotiation.cc:769] Sending connection context
0201 03:08:50.239860 (+ 72us) client_negotiation.cc:240] Negotiation successful
0201 03:08:50.239934 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":58,"threads_started":1}
I20260201 03:08:50.240073 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.233193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:41909 (local address 127.28.181.62:32887)
0201 03:08:50.233233 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.233235 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.233383 (+ 148us) server_negotiation.cc:408] Connection header received
0201 03:08:50.233733 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.233734 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.233742 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.233801 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.234388 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.234903 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.235477 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.235580 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.237651 (+ 2071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.237654 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.237654 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.237661 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.239067 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.239365 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.239366 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.239366 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.239391 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.239548 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.239549 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.239549 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.239654 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.239716 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.239895 (+ 179us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.239976 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260201 03:08:50.240612 3456 heartbeater.cc:344] Connected to a master server at 127.28.181.62:32887
I20260201 03:08:50.240698 3456 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:50.240888 3456 heartbeater.cc:507] Master 127.28.181.62:32887 requested a full tablet report, sending...
I20260201 03:08:50.241211 3127 ts_manager.cc:194] Registered new tserver with Master: b289cc29abbe4431827d063c9b6930ac (127.28.181.2:33523)
I20260201 03:08:50.241540 3127 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:41909
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:08:50.270452 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:32887
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:50.333634 3463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:50.333757 3463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:50.333809 3463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:50.334983 3463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:50.335044 3463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:50.335080 3463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:50.336371 3463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:50.336565 3463 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:50.336719 3463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:50.338713 3469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:50.338758 3468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:50.338775 3463 server_base.cc:1060] running on GCE node
W20260201 03:08:50.338759 3471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:50.338976 3463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:50.339116 3463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:50.340224 3463 hybrid_clock.cc:648] HybridClock initialized: now 1769915330340216 us; error 22 us; skew 500 ppm
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.342010 3463 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:08:50.342444 3463 webserver.cc:492] Webserver started at http://127.28.181.3:45737/ using document root <none> and password file <none>
I20260201 03:08:50.342597 3463 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:50.342643 3463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:50.342743 3463 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:50.343446 3463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "819d43e8a59e4c08ae124be82e2cf247"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "cbb0b2816921e25452a2173acb30630e"
server_key_iv: "d5ebfcc0675de432815e8d7708b70c7b"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.343699 3463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "819d43e8a59e4c08ae124be82e2cf247"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "cbb0b2816921e25452a2173acb30630e"
server_key_iv: "d5ebfcc0675de432815e8d7708b70c7b"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.344637 3463 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:50.345166 3478 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.345307 3463 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:50.345379 3463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "819d43e8a59e4c08ae124be82e2cf247"
format_stamp: "Formatted at 2026-02-01 03:08:50 on dist-test-slave-ck1n"
server_key: "cbb0b2816921e25452a2173acb30630e"
server_key_iv: "d5ebfcc0675de432815e8d7708b70c7b"
server_key_version: "encryptionkey@0"
I20260201 03:08:50.345441 3463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:50.362751 3463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:50.363929 3463 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:50.364053 3463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:50.364217 3463 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:50.364456 3463 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:50.364502 3463 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.364534 3463 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:50.364560 3463 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:50.369104 3463 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:42149
I20260201 03:08:50.369139 3591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:42149 every 8 connection(s)
I20260201 03:08:50.369344 3463 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:50.373178 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3463
I20260201 03:08:50.373272 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:50.373423 29396 external_mini_cluster.cc:1468] Setting key e19a98ab430bc87e78883d10e11a4924
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:08:50.375778 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.369883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:50415 (local address 127.28.181.62:32887)
0201 03:08:50.369959 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.369961 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.370098 (+ 137us) server_negotiation.cc:408] Connection header received
0201 03:08:50.370461 (+ 363us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.370462 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.370472 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.370529 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.371079 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.371490 (+ 411us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.371965 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.372130 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.373949 (+ 1819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.373952 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.373952 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.373959 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.374999 (+ 1040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.375279 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.375280 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.375280 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.375296 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.375423 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.375423 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.375424 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.375484 (+ 60us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.375520 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.375676 (+ 156us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.375726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:08:50.375878 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.369873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:32887 (local address 127.28.181.3:50415)
0201 03:08:50.370008 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:50.370028 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:50.370380 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:50.370569 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:50.370574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:50.370792 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:50.370995 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.370997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.371564 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.371565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:50.371882 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:50.371883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.371998 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.372631 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:50.372643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:50.373870 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:50.375056 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.375060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.375062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.375224 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.375343 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:50.375343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:50.375343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:50.375385 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:50.375522 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:50.375524 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:50.375599 (+ 75us) client_negotiation.cc:769] Sending connection context
0201 03:08:50.375667 (+ 68us) client_negotiation.cc:240] Negotiation successful
0201 03:08:50.375741 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260201 03:08:50.376230 3592 heartbeater.cc:344] Connected to a master server at 127.28.181.62:32887
I20260201 03:08:50.376293 3592 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:50.376399 3592 heartbeater.cc:507] Master 127.28.181.62:32887 requested a full tablet report, sending...
I20260201 03:08:50.376634 3127 ts_manager.cc:194] Registered new tserver with Master: 819d43e8a59e4c08ae124be82e2cf247 (127.28.181.3:42149)
I20260201 03:08:50.376988 3127 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:50415
I20260201 03:08:50.384900 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/test-admin.keytab.
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.395416 29396 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260201 03:08:50.396672 3599 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:50.396071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.28.181.1:41795)
0201 03:08:50.396212 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.396217 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.396226 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:50.396244 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.396245 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.396256 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.396306 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.396599 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57264: BlockingRecv error: recv got EOF from 127.0.0.1:57264 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260201 03:08:50.396699 29396 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc' of type FILE
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.397514 29396 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.398278 29396 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:50.404242 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.398733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.28.181.1:41795)
0201 03:08:50.398791 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.398792 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.398803 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:08:50.398877 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.398878 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.398894 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.398952 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.399395 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.399968 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.400460 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.400649 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.402102 (+ 1453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.402111 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.402111 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.402118 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.403349 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.403640 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.403642 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.403643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.403677 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.403807 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.403808 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.403808 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.403888 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.404029 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.404066 (+ 37us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.404141 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260201 03:08:50.405664 3599 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:50.405173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.28.181.1:41795)
0201 03:08:50.405220 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.405222 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.405278 (+ 56us) server_negotiation.cc:408] Connection header received
0201 03:08:50.405348 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.405349 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.405358 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.405402 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.405626 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57278: BlockingRecv error: recv got EOF from 127.0.0.1:57278 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260201 03:08:50.405733 29396 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc' of type FILE
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.406528 29396 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.407264 29396 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:50.412186 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.407674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.28.181.1:41795)
0201 03:08:50.407719 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.407721 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.407738 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:08:50.407777 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.407778 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.407784 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.407819 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.408183 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.408535 (+ 352us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.408981 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.409073 (+ 92us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.410533 (+ 1460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.410535 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.410535 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.410539 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.411527 (+ 988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.411744 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.411744 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.411744 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.411760 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.411863 (+ 103us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.411864 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.411864 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.411921 (+ 57us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.411965 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.412032 (+ 67us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.412078 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260201 03:08:50.413461 3599 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:08:50.412987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.28.181.1:41795)
0201 03:08:50.413048 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.413049 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.413097 (+ 48us) server_negotiation.cc:408] Connection header received
0201 03:08:50.413166 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.413167 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.413174 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.413220 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.413426 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57290: BlockingRecv error: recv got EOF from 127.0.0.1:57290 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260201 03:08:50.413530 29396 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769915279059265-29396-0/krb5kdc/krb5cc' of type FILE
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.414317 29396 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:50.415019 29396 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.1@KRBTEST.COM
I20260201 03:08:50.419703 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:50.415388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.28.181.1:41795)
0201 03:08:50.415432 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:50.415433 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:50.415450 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:08:50.415490 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:50.415491 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:50.415514 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:50.415552 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:50.415935 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.416260 (+ 325us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:50.416638 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:50.416767 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:50.418136 (+ 1369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:50.418138 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:50.418138 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:50.418142 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:50.419060 (+ 918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.419278 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.419278 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.419278 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.419292 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:50.419384 (+ 92us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:50.419385 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:50.419385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:50.419427 (+ 42us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:50.419472 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:50.419544 (+ 72us) server_negotiation.cc:300] Negotiation successful
0201 03:08:50.419589 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:50.420222 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3191
I20260201 03:08:50.424573 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3327
I20260201 03:08:50.428059 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3463
I20260201 03:08:50.431646 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3095
2026-02-01T03:08:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2421 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3612](info): set up 2 sockets
Feb 01 03:08:50 dist-test-slave-ck1n krb5kdc[3612](info): commencing operation
krb5kdc: starting...
W20260201 03:08:52.141847 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.690s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:08:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:08:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:52.243880 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:46111
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:52.307850 3628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:52.308001 3628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:52.308034 3628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:52.309206 3628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:52.309262 3628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:52.309284 3628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:52.309321 3628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:52.309347 3628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:52.310693 3628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:52.310894 3628 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:52.311053 3628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:52.313287 3633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.313294 3634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.313371 3628 server_base.cc:1060] running on GCE node
W20260201 03:08:52.313292 3636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.313607 3628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:52.313840 3628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:52.314970 3628 hybrid_clock.cc:648] HybridClock initialized: now 1769915332314951 us; error 33 us; skew 500 ppm
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:52.316735 3628 init.cc:377] Logged in from keytab as oryx/127.28.181.62@KRBTEST.COM (short username oryx)
I20260201 03:08:52.317099 3628 webserver.cc:492] Webserver started at http://127.28.181.62:33577/ using document root <none> and password file <none>
I20260201 03:08:52.317224 3628 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.317250 3628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.317308 3628 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:52.318045 3628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "bcb9810ed2544bbb94c0263ac80809a9"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "f2f7985ca021a2e0d8d98e0dee14c475"
server_key_iv: "7816f7959b1e7b243049f517fb5d56bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.318266 3628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "bcb9810ed2544bbb94c0263ac80809a9"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "f2f7985ca021a2e0d8d98e0dee14c475"
server_key_iv: "7816f7959b1e7b243049f517fb5d56bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.319167 3628 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260201 03:08:52.319665 3643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.319813 3628 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.319871 3628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "bcb9810ed2544bbb94c0263ac80809a9"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "f2f7985ca021a2e0d8d98e0dee14c475"
server_key_iv: "7816f7959b1e7b243049f517fb5d56bb"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.319911 3628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.330869 3628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:52.332077 3628 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:52.332154 3628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:52.335189 3628 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:46111
I20260201 03:08:52.335244 3695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:46111 every 8 connection(s)
I20260201 03:08:52.335464 3628 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:08:52.335963 3696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:52.336536 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3628
I20260201 03:08:52.336627 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:08:52.336802 29396 external_mini_cluster.cc:1468] Setting key d8ddb2768a0b88caf2f3a427c43eee5f
I20260201 03:08:52.338428 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Bootstrap starting.
I20260201 03:08:52.338953 3696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:52.339164 3696 log.cc:826] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:52.339709 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: No bootstrap required, opened a new log
I20260201 03:08:52.340680 3696 raft_consensus.cc:359] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } }
I20260201 03:08:52.340768 3696 raft_consensus.cc:385] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:52.340785 3696 raft_consensus.cc:740] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb9810ed2544bbb94c0263ac80809a9, State: Initialized, Role: FOLLOWER
I20260201 03:08:52.340881 3696 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } }
I20260201 03:08:52.340937 3696 raft_consensus.cc:399] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:08:52.340955 3696 raft_consensus.cc:493] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:08:52.341012 3696 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 0 FOLLOWER]: Advancing to term 1
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:52.341475 3696 raft_consensus.cc:515] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } }
I20260201 03:08:52.341557 3696 leader_election.cc:304] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcb9810ed2544bbb94c0263ac80809a9; no voters:
I20260201 03:08:52.341713 3696 leader_election.cc:290] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:08:52.341742 3701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:52.341848 3701 raft_consensus.cc:697] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 LEADER]: Becoming Leader. State: Replica: bcb9810ed2544bbb94c0263ac80809a9, State: Running, Role: LEADER
I20260201 03:08:52.341926 3696 sys_catalog.cc:565] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:52.341914 3701 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } }
I20260201 03:08:52.342168 3701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } }
I20260201 03:08:52.342222 3701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:52.342351 3701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb9810ed2544bbb94c0263ac80809a9. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } }
I20260201 03:08:52.342410 3701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:52.342550 3705 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:52.343009 3705 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:52.345328 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.337209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.28.181.62:46111)
0201 03:08:52.337362 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.337371 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.337389 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:08:52.337706 (+ 317us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.337712 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.337754 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.337911 (+ 157us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.338460 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.339402 (+ 942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.339914 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.340128 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.341827 (+ 1699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.341833 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.341834 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.341852 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.344110 (+ 2258us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.344436 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.344439 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.344439 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.344488 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.344657 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.344658 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.344658 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.344842 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.344982 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.345080 (+ 98us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.345182 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260201 03:08:52.345329 3705 catalog_manager.cc:1357] Generated new cluster ID: b4e384d9a33d4269a52223d002ce4478
I20260201 03:08:52.345391 3705 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:52.365164 3705 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:08:52.365571 3705 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:52.372653 3705 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Generated new TSK 0
I20260201 03:08:52.372783 3705 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:52.415994 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46111
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:52.479094 3724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:52.479219 3724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:52.479238 3724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:52.480402 3724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:52.480438 3724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:52.480459 3724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:52.481848 3724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:52.482009 3724 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:52.482169 3724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:52.484105 3730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.484179 3724 server_base.cc:1060] running on GCE node
W20260201 03:08:52.484118 3729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.484125 3732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.484387 3724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:52.484570 3724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:52.485703 3724 hybrid_clock.cc:648] HybridClock initialized: now 1769915332485692 us; error 22 us; skew 500 ppm
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:52.487561 3724 init.cc:377] Logged in from keytab as oryx/127.28.181.1@KRBTEST.COM (short username oryx)
I20260201 03:08:52.487985 3724 webserver.cc:492] Webserver started at http://127.28.181.1:36089/ using document root <none> and password file <none>
I20260201 03:08:52.488178 3724 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.488225 3724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.488328 3724 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:52.489044 3724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "81843fd47754456a914971183b283442"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "ab666e3fe9db79f0d9675371d557b430"
server_key_iv: "b196b6e14b1497714e1c052efc1be40e"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.489323 3724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "81843fd47754456a914971183b283442"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "ab666e3fe9db79f0d9675371d557b430"
server_key_iv: "b196b6e14b1497714e1c052efc1be40e"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.490366 3724 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:52.490906 3739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.491047 3724 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:52.491125 3724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "81843fd47754456a914971183b283442"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "ab666e3fe9db79f0d9675371d557b430"
server_key_iv: "b196b6e14b1497714e1c052efc1be40e"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.491196 3724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.503726 3724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:52.507366 3724 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:52.507496 3724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:52.507666 3724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:52.507891 3724 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:52.507954 3724 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.507989 3724 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:52.508015 3724 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.513618 3724 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:46441
I20260201 03:08:52.513671 3852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:46441 every 8 connection(s)
I20260201 03:08:52.513926 3724 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:52.518790 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3724
I20260201 03:08:52.519119 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:08:52.519308 29396 external_mini_cluster.cc:1468] Setting key 814c4415c3f153daf34d795bff7d9e1a
I20260201 03:08:52.520690 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.514409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:47067)
0201 03:08:52.514517 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.514541 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.514872 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.515019 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.515021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.515240 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.515457 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.515459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.515967 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.515967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.516267 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.516267 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.516347 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.516948 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.516959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.518156 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.519703 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.519707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.519709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.519931 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.520087 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.520088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.520088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.520146 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.520350 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.520353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.520430 (+ 77us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.520506 (+ 76us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.520582 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":63,"threads_started":1}
I20260201 03:08:52.520746 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.514435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:47067 (local address 127.28.181.62:46111)
0201 03:08:52.514468 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.514470 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.514602 (+ 132us) server_negotiation.cc:408] Connection header received
0201 03:08:52.514933 (+ 331us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.514934 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.514943 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.514988 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.515515 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.515906 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.516321 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.516409 (+ 88us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.518242 (+ 1833us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.518247 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.518247 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.518252 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.519643 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.519998 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.519999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.520000 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.520022 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.520199 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.520200 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.520200 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.520299 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.520358 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.520541 (+ 183us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.520626 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:08:52.521386 3853 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:52.521446 3853 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:52.521597 3853 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:52.522020 3660 ts_manager.cc:194] Registered new tserver with Master: 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:52.522581 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:47067
WARNING: no policy specified for oryx/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:52.553694 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46111
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:52.617977 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:52.618117 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:52.618135 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:52.619277 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:52.619308 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:52.619329 3860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:52.620541 3860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:52.620677 3860 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:52.620775 3860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:52.622639 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.622654 3860 server_base.cc:1060] running on GCE node
W20260201 03:08:52.622707 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.622637 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.622936 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:52.623099 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:52.624224 3860 hybrid_clock.cc:648] HybridClock initialized: now 1769915332624221 us; error 35 us; skew 500 ppm
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:52.625939 3860 init.cc:377] Logged in from keytab as oryx/127.28.181.2@KRBTEST.COM (short username oryx)
I20260201 03:08:52.626395 3860 webserver.cc:492] Webserver started at http://127.28.181.2:34417/ using document root <none> and password file <none>
I20260201 03:08:52.626559 3860 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.626605 3860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.626703 3860 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:52.627441 3860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "3b429af07ea346829b21aec63290f890"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "3297de39fe791d466c17421aba5b54ce"
server_key_iv: "71327650196524e32b2db520c4919fe3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.627698 3860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "3b429af07ea346829b21aec63290f890"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "3297de39fe791d466c17421aba5b54ce"
server_key_iv: "71327650196524e32b2db520c4919fe3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.628639 3860 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:52.629166 3875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.629305 3860 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:52.629374 3860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "3b429af07ea346829b21aec63290f890"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "3297de39fe791d466c17421aba5b54ce"
server_key_iv: "71327650196524e32b2db520c4919fe3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.629437 3860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.647569 3860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:52.648806 3860 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:52.648932 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:52.649097 3860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:52.649339 3860 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:52.649384 3860 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.649410 3860 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:52.649433 3860 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.654506 3860 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:34343
I20260201 03:08:52.654778 3860 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:08:52.655128 3988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:34343 every 8 connection(s)
I20260201 03:08:52.656519 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3860
I20260201 03:08:52.656594 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:08:52.656749 29396 external_mini_cluster.cc:1468] Setting key 18bdf413d453376c463d683090717ee4
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:52.662340 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.655392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:39995 (local address 127.28.181.62:46111)
0201 03:08:52.655447 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.655449 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.655552 (+ 103us) server_negotiation.cc:408] Connection header received
0201 03:08:52.655973 (+ 421us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.655974 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.655997 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.656053 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.656761 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.657377 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.657883 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.658069 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.660091 (+ 2022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.660095 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.660095 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.660101 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.661234 (+ 1133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.661572 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.661573 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.661573 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.661601 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.661802 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.661802 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.661803 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.661888 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.661957 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.662185 (+ 228us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.662261 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:08:52.662427 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.655347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:39995)
0201 03:08:52.655453 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.655494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.655892 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.656097 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.656100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.656387 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.656675 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.656678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.657466 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.657467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.657805 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.657807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.657892 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.658718 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.658737 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.660013 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.661322 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.661327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.661330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.661498 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.661656 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.661656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.661657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.661713 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.661948 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.661952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.662049 (+ 97us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.662166 (+ 117us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.662268 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260201 03:08:52.662846 3989 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:52.662933 3989 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:52.663081 3989 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:52.663381 3660 ts_manager.cc:194] Registered new tserver with Master: 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
WARNING: no policy specified for oryx/127.28.181.3@KRBTEST.COM; defaulting to no policy
I20260201 03:08:52.663820 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:39995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:52.692087 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46111
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:52.754752 3996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:52.754920 3996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:52.754953 3996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:52.756124 3996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:52.756181 3996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:52.756218 3996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:52.757563 3996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:52.757750 3996 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:52.757927 3996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:52.759761 4001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.759769 4002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.759768 4004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.759779 3996 server_base.cc:1060] running on GCE node
I20260201 03:08:52.760047 3996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:52.760231 3996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:52.761363 3996 hybrid_clock.cc:648] HybridClock initialized: now 1769915332761341 us; error 45 us; skew 500 ppm
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:52.763221 3996 init.cc:377] Logged in from keytab as oryx/127.28.181.3@KRBTEST.COM (short username oryx)
I20260201 03:08:52.763613 3996 webserver.cc:492] Webserver started at http://127.28.181.3:34543/ using document root <none> and password file <none>
I20260201 03:08:52.763772 3996 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.763818 3996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.763921 3996 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:52.764688 3996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "d1801f0dfc7c44da984786c68e85c06f"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "4c034fcf90b85397b0f645a1daf162f9"
server_key_iv: "f78be5cc7da415fe6c6cd22fecad19d3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.764954 3996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "d1801f0dfc7c44da984786c68e85c06f"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "4c034fcf90b85397b0f645a1daf162f9"
server_key_iv: "f78be5cc7da415fe6c6cd22fecad19d3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.765942 3996 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:08:52.766494 4011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.766630 3996 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.766708 3996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "d1801f0dfc7c44da984786c68e85c06f"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "4c034fcf90b85397b0f645a1daf162f9"
server_key_iv: "f78be5cc7da415fe6c6cd22fecad19d3"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.766770 3996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.779371 3996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:52.780506 3996 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:52.780622 3996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:52.780776 3996 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:52.780995 3996 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:52.781042 3996 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.781075 3996 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:52.781100 3996 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.785894 3996 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:41021
I20260201 03:08:52.785933 4124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:41021 every 8 connection(s)
I20260201 03:08:52.786139 3996 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:52.792438 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.786616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:50413 (local address 127.28.181.62:46111)
0201 03:08:52.786667 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.786670 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.786837 (+ 167us) server_negotiation.cc:408] Connection header received
0201 03:08:52.787236 (+ 399us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.787238 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.787248 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.787321 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.787934 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.788358 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.788846 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.789018 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.790696 (+ 1678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.790698 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.790699 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.790704 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.791644 (+ 940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.791914 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.791914 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.791914 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.791929 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.792061 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.792061 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.792062 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.792120 (+ 58us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.792157 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.792335 (+ 178us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.792385 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:08:52.792519 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.786603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:50413)
0201 03:08:52.786746 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.786765 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.787152 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.787371 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.787374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.787613 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.787842 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.787843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.788429 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.788430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.788744 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.788745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.788884 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.789523 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.789534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.790629 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.791703 (+ 1074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.791707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.791709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.791857 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.791976 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.791977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.791977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.792027 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.792161 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.792164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.792240 (+ 76us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.792316 (+ 76us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.792392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260201 03:08:52.792891 4125 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:52.792980 4125 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:52.793110 4125 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:52.793414 3660 ts_manager.cc:194] Registered new tserver with Master: d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021)
I20260201 03:08:52.793709 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:50413
I20260201 03:08:52.794764 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 3996
I20260201 03:08:52.794862 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:08:52.795001 29396 external_mini_cluster.cc:1468] Setting key 662965e5ba9279bd9adc6f8bf0db48d3
I20260201 03:08:52.795759 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:52.829443 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.61:40901
--webserver_interface=127.28.181.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:46111,127.28.181.61:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:52.892489 4132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:52.892639 4132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:52.892671 4132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:52.893862 4132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:52.893918 4132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:52.893940 4132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:52.893967 4132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:52.893986 4132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:52.895334 4132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:46111,127.28.181.61:40901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.61:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.181.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
W20260201 03:08:52.895435 4132 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.181.62:46111,127.28.181.61:40901'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260201 03:08:52.895551 4132 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:52.895700 4132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:52.897442 4137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.897483 4140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:52.897449 4138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:52.897585 4132 server_base.cc:1060] running on GCE node
I20260201 03:08:52.897871 4132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:52.898065 4132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:52.899178 4132 hybrid_clock.cc:648] HybridClock initialized: now 1769915332899171 us; error 21 us; skew 500 ppm
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:52.900906 4132 init.cc:377] Logged in from keytab as oryx/127.28.181.61@KRBTEST.COM (short username oryx)
I20260201 03:08:52.901304 4132 webserver.cc:492] Webserver started at http://127.28.181.61:45111/ using document root <none> and password file <none>
I20260201 03:08:52.901463 4132 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.901510 4132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.901610 4132 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:52.902387 4132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/instance:
uuid: "72d8bc3b19e5459caa11176e27435bd1"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "d0021cab6437a59ad59e2cc0934801c2"
server_key_iv: "929485471f5a2f5c6b3f2f3a0e772018"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.902648 4132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal/instance:
uuid: "72d8bc3b19e5459caa11176e27435bd1"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "d0021cab6437a59ad59e2cc0934801c2"
server_key_iv: "929485471f5a2f5c6b3f2f3a0e772018"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.903601 4132 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:52.904140 4147 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.904279 4132 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:52.904345 4132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
uuid: "72d8bc3b19e5459caa11176e27435bd1"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "d0021cab6437a59ad59e2cc0934801c2"
server_key_iv: "929485471f5a2f5c6b3f2f3a0e772018"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.904408 4132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.923931 4132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:52.925452 4132 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:52.925573 4132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:52.928784 4132 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.61:40901
I20260201 03:08:52.928823 4199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.61:40901 every 8 connection(s)
I20260201 03:08:52.929040 4132 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/info.pb
I20260201 03:08:52.929569 4200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:52.930949 4200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } has no permanent_uuid. Determining permanent_uuid...
I20260201 03:08:52.932607 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 4132
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.61@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:52.937547 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.931424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.28.181.62:46111)
0201 03:08:52.931493 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.931495 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.931605 (+ 110us) server_negotiation.cc:408] Connection header received
0201 03:08:52.931894 (+ 289us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.931896 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.931905 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.931958 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.932547 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.933039 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.933486 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.933667 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.935036 (+ 1369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.935039 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.935040 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.935046 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.936382 (+ 1336us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.936751 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.936752 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.936752 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.936772 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.936940 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.936941 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.936941 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.937070 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.937118 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.937369 (+ 251us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.937457 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260201 03:08:52.937575 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.931393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35502)
0201 03:08:52.931525 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.931544 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.931818 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.932001 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.932004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.932198 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.932404 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.932406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.933131 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.933131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.933420 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.933421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.933489 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.933899 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.933912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.934955 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.936458 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.936462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.936464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.936682 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.936826 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.936827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.936827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.936887 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.937130 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.937133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.937249 (+ 116us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.937354 (+ 105us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.937465 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260201 03:08:52.938369 4200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } has no permanent_uuid. Determining permanent_uuid...
I20260201 03:08:52.939939 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.932992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.28.181.61:40901)
0201 03:08:52.933121 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.933127 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.933142 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:08:52.933159 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.933161 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.933178 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.933242 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.933968 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.934491 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.935070 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.935209 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.937434 (+ 2225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.937441 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.937441 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.937446 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.939086 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.939379 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.939381 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.939381 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.939410 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.939532 (+ 122us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.939533 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.939533 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.939618 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.939720 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.939773 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.939841 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.61@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:52.943679 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.938543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38372)
0201 03:08:52.938688 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.938697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.938738 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.938852 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.938852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.939044 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.939179 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.939181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.939957 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.939958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.940350 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.940352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.940409 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.940719 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.940723 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.941564 (+ 841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.943107 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.943108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.943108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.943215 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.943339 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.943340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.943340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.943358 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.943521 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.943522 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.943548 (+ 26us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.943588 (+ 40us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.943636 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260201 03:08:52.943807 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.938590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.28.181.61:40901)
0201 03:08:52.938698 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.938699 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.938727 (+ 28us) server_negotiation.cc:408] Connection header received
0201 03:08:52.938785 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.938785 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.938792 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.938850 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.939245 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.939824 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.940408 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.940532 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.941619 (+ 1087us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.941621 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.941621 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.941625 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.943051 (+ 1426us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.943268 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.943268 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.943269 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.943283 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.943424 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.943424 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.943424 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.943480 (+ 56us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.943519 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.943639 (+ 120us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.943691 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":75,"threads_started":1}
I20260201 03:08:52.944597 4200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Bootstrap starting.
I20260201 03:08:52.944974 4200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:52.945150 4200 log.cc:826] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:52.945614 4200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: No bootstrap required, opened a new log
I20260201 03:08:52.946519 4200 raft_consensus.cc:359] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } }
I20260201 03:08:52.946623 4200 raft_consensus.cc:385] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:52.946671 4200 raft_consensus.cc:740] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8bc3b19e5459caa11176e27435bd1, State: Initialized, Role: FOLLOWER
I20260201 03:08:52.946756 4200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } }
I20260201 03:08:52.946974 4200 sys_catalog.cc:565] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:52.946959 4207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } } }
I20260201 03:08:52.947018 4207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: FOLLOWER
W20260201 03:08:52.948441 4218 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:52.948490 4218 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:52.949520 4132 master_runner.cc:428] Detected that this master 72d8bc3b19e5459caa11176e27435bd1 is joining an existing cluster
I20260201 03:08:52.949551 4132 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.181.61:40901
I20260201 03:08:52.954542 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.950380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38376)
0201 03:08:52.950579 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.950587 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.950641 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.950750 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.950751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.950896 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.951037 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.951038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.951525 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.951526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.951797 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.951798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.951878 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.952221 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.952224 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.952401 (+ 177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.953753 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.953754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.953754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.953904 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.954049 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.954050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.954050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.954072 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.954249 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.954249 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.954278 (+ 29us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.954362 (+ 84us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.954443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":51,"threads_started":1}
I20260201 03:08:52.954577 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.950344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.28.181.61:40901)
0201 03:08:52.950480 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.950481 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.950648 (+ 167us) server_negotiation.cc:408] Connection header received
0201 03:08:52.950683 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.950684 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.950690 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.950710 (+ 20us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.951104 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.951453 (+ 349us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.951870 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.952022 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.952472 (+ 450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.952474 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.952475 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.952480 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.953670 (+ 1190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.953973 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.953973 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.953974 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.953991 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.954118 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.954118 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.954119 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.954196 (+ 77us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.954255 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.954378 (+ 123us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.954486 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":123}
I20260201 03:08:52.972352 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.950512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.28.181.62:46111)
0201 03:08:52.950558 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.950560 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.950568 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:52.950587 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.950588 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.950597 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.950651 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.951481 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.952339 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.953015 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.953155 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.970388 (+ 17233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.970391 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.970391 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.970396 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.971584 (+ 1188us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.971891 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.971892 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.971892 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.971906 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.972029 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.972029 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.972029 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.972088 (+ 59us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.972132 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.972254 (+ 122us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.972303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:52.972397 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.950335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35516)
0201 03:08:52.950476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.950486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.950532 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.950680 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.950681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.950857 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.951393 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.951394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.952425 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.952425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.952930 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.952932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.953190 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.970055 (+ 16865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.970059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.970316 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.971665 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.971666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.971667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.971830 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.971957 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.971958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.971958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.971982 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.972134 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.972135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.972177 (+ 42us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.972236 (+ 59us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.972314 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Feb 01 03:08:52 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.62@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:52.979198 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.974007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.28.181.61:40901)
0201 03:08:52.974039 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.974041 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.974151 (+ 110us) server_negotiation.cc:408] Connection header received
0201 03:08:52.974224 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.974225 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.974232 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.974280 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.974752 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.975112 (+ 360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.975574 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.975738 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.977189 (+ 1451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:52.977191 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:52.977192 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:52.977197 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:52.978325 (+ 1128us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.978642 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.978642 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.978643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.978665 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:52.978795 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:52.978796 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:52.978796 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:52.978861 (+ 65us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:52.978926 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:52.979069 (+ 143us) server_negotiation.cc:300] Negotiation successful
0201 03:08:52.979128 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:08:52.979229 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.973976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38378)
0201 03:08:52.974069 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.974090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.974174 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.974283 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.974285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.974536 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.974675 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.974676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.975200 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:52.975201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:52.975494 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.975495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:52.975577 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:52.975996 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:52.975999 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:52.977110 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:52.978405 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.978411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.978413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.978567 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.978726 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:52.978727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:52.978727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:52.978752 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:52.978920 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:52.978925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:52.978965 (+ 40us) client_negotiation.cc:769] Sending connection context
0201 03:08:52.979049 (+ 84us) client_negotiation.cc:240] Negotiation successful
0201 03:08:52.979137 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":43,"threads_started":1}
I20260201 03:08:52.979583 3660 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.28.181.61:40901: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1"
member_type: NON_VOTER
last_known_addr {
host: "127.28.181.61"
port: 40901
}
attrs {
promote: true
}
}
dest_uuid: "bcb9810ed2544bbb94c0263ac80809a9"
cas_config_opid_index: -1
I20260201 03:08:52.979780 3660 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } }
I20260201 03:08:52.980152 4227 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } }
I20260201 03:08:52.980336 4174 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:52.981040 4174 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260201 03:08:52.981041 4227 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.28.181.61:40901
I20260201 03:08:52.981215 4132 master.cc:561] Master@127.28.181.61:40901 shutting down...
I20260201 03:08:52.981343 4227 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:52.981385 4228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } }
I20260201 03:08:52.981460 4228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
W20260201 03:08:52.981945 3647 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260201 03:08:52.982275 3647 consensus_peers.cc:597] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 -> Peer 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61:40901): Couldn't send request to peer 72d8bc3b19e5459caa11176e27435bd1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260201 03:08:52.983492 4132 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:52.983558 4132 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:52.983613 4132 tablet_replica.cc:333] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: stopping tablet replica
I20260201 03:08:52.995416 4132 master.cc:583] Master@127.28.181.61:40901 shutdown complete.
I20260201 03:08:52.995446 4132 master_runner.cc:305] Clearing existing system tablet
I20260201 03:08:52.995560 4132 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:52.995590 4132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:52.996059 4132 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.996464 4233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.996565 4132 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:52.996613 4132 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:52.996631 4132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
uuid: "72d8bc3b19e5459caa11176e27435bd1"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "d0021cab6437a59ad59e2cc0934801c2"
server_key_iv: "929485471f5a2f5c6b3f2f3a0e772018"
server_key_version: "encryptionkey@0"
I20260201 03:08:52.997035 4132 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:52.998116 4132 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260201 03:08:52.998162 4132 log.cc:1199] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260201 03:08:52.998325 4132 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Deleting consensus metadata
I20260201 03:08:52.998422 4132 master_runner.cc:315] Copying system tablet from 127.28.181.62:46111
I20260201 03:08:52.998901 4132 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: tablet copy: Beginning tablet copy session from remote peer at address 127.28.181.62:46111
I20260201 03:08:53.003269 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.999178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.28.181.62:46111)
0201 03:08:52.999215 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:52.999216 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:52.999283 (+ 67us) server_negotiation.cc:408] Connection header received
0201 03:08:52.999352 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:52.999353 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:52.999360 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:52.999406 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:52.999719 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.000101 (+ 382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.000530 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.000678 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.001158 (+ 480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.001161 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.001161 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.001167 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.002346 (+ 1179us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.002659 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.002660 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.002660 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.002684 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.002838 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.002839 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.002839 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.002919 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.002989 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.003135 (+ 146us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.003198 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:53.003283 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:52.999129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35518)
0201 03:08:52.999213 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:52.999223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:52.999302 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:52.999409 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:52.999409 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:52.999533 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:52.999654 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:52.999655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.000178 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.000179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.000463 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.000464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.000532 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.000896 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.000898 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.001080 (+ 182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.002439 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.002440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.002441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.002580 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.002749 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.002750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.002751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.002784 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.002980 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.002981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.003026 (+ 45us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.003113 (+ 87us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.003199 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":51,"threads_started":1}
I20260201 03:08:53.003648 3680 tablet_copy_service.cc:140] P bcb9810ed2544bbb94c0263ac80809a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 72d8bc3b19e5459caa11176e27435bd1 ({username='oryx', principal='oryx/127.28.181.61@KRBTEST.COM'} at 127.0.0.1:35518)
I20260201 03:08:53.003710 3680 tablet_copy_service.cc:161] P bcb9810ed2544bbb94c0263ac80809a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 72d8bc3b19e5459caa11176e27435bd1 at {username='oryx', principal='oryx/127.28.181.61@KRBTEST.COM'} at 127.0.0.1:35518: session id = 72d8bc3b19e5459caa11176e27435bd1-00000000000000000000000000000000
I20260201 03:08:53.004135 3680 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260201 03:08:53.004541 4132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260201 03:08:53.005177 4132 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: tablet copy: Starting download of 0 data blocks...
I20260201 03:08:53.005285 4132 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: tablet copy: Starting download of 1 WAL segments...
I20260201 03:08:53.006011 4132 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260201 03:08:53.006641 3680 tablet_copy_service.cc:342] P bcb9810ed2544bbb94c0263ac80809a9: Request end of tablet copy session 72d8bc3b19e5459caa11176e27435bd1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.181.61@KRBTEST.COM'} at 127.0.0.1:35518
I20260201 03:08:53.006696 3680 tablet_copy_service.cc:434] P bcb9810ed2544bbb94c0263ac80809a9: ending tablet copy session 72d8bc3b19e5459caa11176e27435bd1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 72d8bc3b19e5459caa11176e27435bd1
W20260201 03:08:53.007966 4132 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260201 03:08:53.008548 4132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:53.009369 4240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:53.009403 4243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.009577 4132 server_base.cc:1060] running on GCE node
W20260201 03:08:53.009867 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.009989 4132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:53.010099 4132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:53.011234 4132 hybrid_clock.cc:648] HybridClock initialized: now 1769915333011238 us; error 32 us; skew 500 ppm
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:53.012285 4132 init.cc:377] Logged in from keytab as oryx/127.28.181.61@KRBTEST.COM (short username oryx)
I20260201 03:08:53.012502 4132 webserver.cc:492] Webserver started at http://127.28.181.61:34341/ using document root <none> and password file <none>
I20260201 03:08:53.012609 4132 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:53.012647 4132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:53.013130 4132 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.013542 4250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.013650 4132 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.013706 4132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
uuid: "72d8bc3b19e5459caa11176e27435bd1"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "d0021cab6437a59ad59e2cc0934801c2"
server_key_iv: "929485471f5a2f5c6b3f2f3a0e772018"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.013760 4132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:53.023206 4132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:53.024437 4132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:53.027081 4132 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.61:40901
I20260201 03:08:53.027302 4302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.61:40901 every 8 connection(s)
I20260201 03:08:53.027377 4132 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-1/data/info.pb
I20260201 03:08:53.027989 4303 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:08:53.028378 4303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Bootstrap starting.
I20260201 03:08:53.030499 4303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:08:53.030680 4303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: Bootstrap complete.
I20260201 03:08:53.030843 4303 raft_consensus.cc:359] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } }
I20260201 03:08:53.030906 4303 raft_consensus.cc:740] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72d8bc3b19e5459caa11176e27435bd1, State: Initialized, Role: LEARNER
I20260201 03:08:53.030936 4303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } }
I20260201 03:08:53.031067 4303 sys_catalog.cc:565] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:53.031073 4306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } }
I20260201 03:08:53.031177 4306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:08:53.032347 4317 catalog_manager.cc:1269] Loaded cluster ID: b4e384d9a33d4269a52223d002ce4478
I20260201 03:08:53.032390 4317 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: loading cluster ID for follower catalog manager: success
I20260201 03:08:53.033217 4317 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: acquiring CA information for follower catalog manager: success
I20260201 03:08:53.033445 4317 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260201 03:08:53.046407 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.041162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.28.181.61:40901)
0201 03:08:53.041249 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.041250 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.041261 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:08:53.041275 (+ 14us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.041275 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.041282 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.041351 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.042092 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.042679 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.043340 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.043430 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.044359 (+ 929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.044362 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.044362 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.044367 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.045564 (+ 1197us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.045910 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.045911 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.045911 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.045927 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.046067 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.046067 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.046067 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.046129 (+ 62us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.046182 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.046301 (+ 119us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.046351 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.28.181.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:08:53.085494 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.60:38849
--webserver_interface=127.28.181.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:53.150808 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:53.150978 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:53.150996 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:53.152226 4323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:08:53.152261 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:53.152273 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:53.152284 4323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:08:53.152294 4323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:08:53.153589 4323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.60:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.181.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:53.153772 4323 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:53.153983 4323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:53.155772 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.155819 4323 server_base.cc:1060] running on GCE node
W20260201 03:08:53.155773 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:53.155769 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.156165 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:53.156374 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:53.157513 4323 hybrid_clock.cc:648] HybridClock initialized: now 1769915333157500 us; error 43 us; skew 500 ppm
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:53.159562 4323 init.cc:377] Logged in from keytab as oryx/127.28.181.60@KRBTEST.COM (short username oryx)
I20260201 03:08:53.159987 4323 webserver.cc:492] Webserver started at http://127.28.181.60:33567/ using document root <none> and password file <none>
I20260201 03:08:53.160108 4323 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:53.160136 4323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:53.160197 4323 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:08:53.160866 4323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/instance:
uuid: "0867071355a643f59e50d5390fec86b1"
format_stamp: "Formatted at 2026-02-01 03:08:53 on dist-test-slave-ck1n"
server_key: "47ad93410187cd026872bbb1436f2ee4"
server_key_iv: "cb08fb4c737f5e8b5196e01f955dc3da"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.161082 4323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal/instance:
uuid: "0867071355a643f59e50d5390fec86b1"
format_stamp: "Formatted at 2026-02-01 03:08:53 on dist-test-slave-ck1n"
server_key: "47ad93410187cd026872bbb1436f2ee4"
server_key_iv: "cb08fb4c737f5e8b5196e01f955dc3da"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.162029 4323 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:08:53.162532 4338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.162673 4323 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:53.162732 4323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
uuid: "0867071355a643f59e50d5390fec86b1"
format_stamp: "Formatted at 2026-02-01 03:08:53 on dist-test-slave-ck1n"
server_key: "47ad93410187cd026872bbb1436f2ee4"
server_key_iv: "cb08fb4c737f5e8b5196e01f955dc3da"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.162770 4323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:53.187846 4323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:53.189152 4323 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:53.189232 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:53.192308 4323 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.60:38849
I20260201 03:08:53.192351 4390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.60:38849 every 8 connection(s)
I20260201 03:08:53.192565 4323 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/info.pb
I20260201 03:08:53.193076 4391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:53.194547 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.60@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:53.198541 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 4323
I20260201 03:08:53.201081 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.195150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.28.181.62:46111)
0201 03:08:53.195205 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.195207 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.195215 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:53.195456 (+ 241us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.195456 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.195464 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.195509 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.196041 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.196440 (+ 399us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.196844 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.196953 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.198622 (+ 1669us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.198626 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.198627 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.198633 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.200048 (+ 1415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.200360 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.200362 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.200362 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.200385 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.200541 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.200541 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.200542 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.200626 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.200691 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.200917 (+ 226us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.200992 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:08:53.201146 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.194993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35520)
0201 03:08:53.195126 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.195151 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.195388 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.195560 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.195562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.195756 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.195975 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.195978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.196501 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.196501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.196789 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.196790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.196858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.197249 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.197262 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.198539 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.200116 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.200120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.200122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.200293 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.200433 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.200434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.200435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.200484 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.200682 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.200686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.200794 (+ 108us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.200899 (+ 105us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.200998 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260201 03:08:53.201644 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.60@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:53.206518 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.198956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.28.181.60:38849)
0201 03:08:53.199078 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.199084 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.199100 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:08:53.199115 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.199117 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.199130 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.199179 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.199964 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.200501 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.200983 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.201181 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.203615 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.203621 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.203622 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.203627 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.205488 (+ 1861us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.205812 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.205815 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.205816 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.205852 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.206014 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.206014 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.206015 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.206105 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.206243 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.206322 (+ 79us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.206396 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260201 03:08:53.207494 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.201919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38420)
0201 03:08:53.202049 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.202065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.202121 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.202245 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.202245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.202394 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.202576 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.202578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.203178 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.203180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.203520 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.203521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.203590 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.203942 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.203945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.205166 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.206901 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.206902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.206902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.207068 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.207186 (+ 118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.207187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.207187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.207204 (+ 17us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.207332 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.207333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.207357 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.207396 (+ 39us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.207448 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260201 03:08:53.207542 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.201991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.28.181.61:40901)
0201 03:08:53.202046 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.202048 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.202106 (+ 58us) server_negotiation.cc:408] Connection header received
0201 03:08:53.202181 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.202182 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.202194 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.202241 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.202645 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.203077 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.203591 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.203751 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.205242 (+ 1491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.205244 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.205245 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.205250 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.206838 (+ 1588us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.207130 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.207131 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.207131 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.207149 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.207233 (+ 84us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.207233 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.207233 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.207297 (+ 64us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.207339 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.207409 (+ 70us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.207467 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:08:53.207844 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.60@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
I20260201 03:08:53.212301 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.208056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48794)
0201 03:08:53.208128 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.208136 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.208181 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.208275 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.208275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.208362 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.208446 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.208447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.208973 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.208974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.209251 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.209251 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.209300 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.209636 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.209639 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.210591 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.211765 (+ 1174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.211765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.211765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.211877 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.211993 (+ 116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.211993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.211993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.212011 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.212135 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.212135 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.212160 (+ 25us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.212200 (+ 40us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.212258 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260201 03:08:53.212427 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.208104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.28.181.60:38849)
0201 03:08:53.208146 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.208147 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.208172 (+ 25us) server_negotiation.cc:408] Connection header received
0201 03:08:53.208216 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.208217 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.208223 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.208259 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.208496 (+ 237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.208917 (+ 421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.209311 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.209418 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.210654 (+ 1236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.210657 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.210657 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.210661 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.211704 (+ 1043us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.211931 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.211931 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.211931 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.211946 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.212041 (+ 95us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.212041 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.212041 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.212094 (+ 53us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.212128 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.212211 (+ 83us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.212301 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":27}
I20260201 03:08:53.213245 4391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Bootstrap starting.
I20260201 03:08:53.213758 4391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:53.214021 4391 log.cc:826] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:53.214536 4391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: No bootstrap required, opened a new log
I20260201 03:08:53.215423 4391 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } }
I20260201 03:08:53.215519 4391 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:53.215572 4391 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0867071355a643f59e50d5390fec86b1, State: Initialized, Role: FOLLOWER
I20260201 03:08:53.215654 4391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } }
I20260201 03:08:53.215867 4391 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:53.215879 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } } }
I20260201 03:08:53.215945 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260201 03:08:53.217114 4407 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:08:53.217177 4407 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:08:53.218636 4323 master_runner.cc:428] Detected that this master 0867071355a643f59e50d5390fec86b1 is joining an existing cluster
I20260201 03:08:53.218677 4323 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.181.60:38849
I20260201 03:08:53.243479 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.28.181.61:40901)
0201 03:08:53.219530 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.219531 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.219708 (+ 177us) server_negotiation.cc:408] Connection header received
0201 03:08:53.219784 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.219785 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.219792 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.219829 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.238457 (+ 18628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239082 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239803 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239969 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.240906 (+ 937us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.240909 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.240909 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.240914 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.242451 (+ 1537us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.242905 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.242906 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.242907 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.242929 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.243068 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.243069 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.243069 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.243150 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.243184 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.243337 (+ 153us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.243403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260201 03:08:53.243609 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.28.181.60:38849)
0201 03:08:53.219781 (+ 26us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.219782 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.225968 (+ 6186us) server_negotiation.cc:408] Connection header received
0201 03:08:53.226011 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.226011 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.226018 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.226064 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.238464 (+ 12400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239021 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239665 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239797 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.240672 (+ 875us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.240674 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.240674 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.240679 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.242432 (+ 1753us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.243050 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.243051 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.243051 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.243071 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.243204 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.243204 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.243204 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.243281 (+ 77us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.243312 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.243468 (+ 156us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.243530 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:08:53.243721 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38428)
0201 03:08:53.219654 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.219662 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.219726 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.219883 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.219883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.238077 (+ 18194us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.238367 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.238368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239156 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.239718 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.239719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.240007 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.240375 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.240379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.240801 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.242533 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.242534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.242535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.242826 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.242985 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.242985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.242985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.243008 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.243220 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.243221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.243252 (+ 31us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.243563 (+ 311us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.243641 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260201 03:08:53.243894 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48796)
0201 03:08:53.225893 (+ 6249us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.225903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.225962 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.226088 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.226089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.238127 (+ 12038us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.238382 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.238384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239094 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.239596 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.239597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239837 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.240219 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.240223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.240594 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.242514 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.242515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.242515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.242771 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.243127 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.243128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.243128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.243151 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.243345 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.243346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.243374 (+ 28us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.243698 (+ 324us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.243785 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6236,"thread_start_us":60,"threads_started":1}
I20260201 03:08:53.244920 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.28.181.62:46111)
0201 03:08:53.219523 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.219525 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.219628 (+ 103us) server_negotiation.cc:408] Connection header received
0201 03:08:53.219716 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.219717 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.219725 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.219751 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.238457 (+ 18706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239073 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239803 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.240380 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.241449 (+ 1069us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.241452 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.241452 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.241458 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.243591 (+ 2133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.244354 (+ 763us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.244355 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.244356 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.244375 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.244509 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.244510 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.244510 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.244585 (+ 75us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.244646 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.244764 (+ 118us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.244823 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:53.244926 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.219449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35528)
0201 03:08:53.219572 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.219581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.219652 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.219807 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.219807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.238107 (+ 18300us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.238363 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.238365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.239158 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.239158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.239718 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.239719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.240422 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.240983 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.240987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.241362 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.244145 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.244146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.244146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.244280 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.244438 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.244438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.244439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.244462 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.244639 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.244639 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.244671 (+ 32us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.244744 (+ 73us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.244823 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":85,"threads_started":1}
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915332, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.62@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
I20260201 03:08:53.251293 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.246561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48800)
0201 03:08:53.246601 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.246608 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.246640 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.246771 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.246771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.246894 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.246989 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.246990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.247501 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.247502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.247781 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.247782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.247848 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.248207 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.248210 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.249144 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.250662 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.250663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.250663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.250774 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.250913 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.250914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.250914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.250933 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.251106 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.251107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.251133 (+ 26us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.251176 (+ 43us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.251246 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260201 03:08:53.251333 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.246603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.28.181.60:38849)
0201 03:08:53.246630 (+ 27us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.246631 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.246639 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:53.246698 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.246699 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.246707 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.246767 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.247086 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.247420 (+ 334us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.247845 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.247998 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.249215 (+ 1217us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.249217 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.249217 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.249223 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.250598 (+ 1375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.250836 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.250836 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.250837 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.250854 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.250969 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.250969 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.250969 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.251058 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.251110 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.251194 (+ 84us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.251265 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:08:53.251756 3660 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.28.181.60:38849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0867071355a643f59e50d5390fec86b1"
member_type: NON_VOTER
last_known_addr {
host: "127.28.181.60"
port: 38849
}
attrs {
promote: true
}
}
dest_uuid: "bcb9810ed2544bbb94c0263ac80809a9"
cas_config_opid_index: 5
I20260201 03:08:53.251953 3660 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } }
I20260201 03:08:53.252228 4227 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0867071355a643f59e50d5390fec86b1 (127.28.181.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.252415 4365 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:53.252916 4227 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.28.181.60:38849
I20260201 03:08:53.252983 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.253051 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:53.253384 4323 master.cc:561] Master@127.28.181.60:38849 shutting down...
I20260201 03:08:53.253418 4365 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260201 03:08:53.253613 4229 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260201 03:08:53.254252 3646 consensus_peers.cc:597] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 -> Peer 0867071355a643f59e50d5390fec86b1 (127.28.181.60:38849): Couldn't send request to peer 0867071355a643f59e50d5390fec86b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260201 03:08:53.256207 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.253232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38444)
0201 03:08:53.253386 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.253395 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.253429 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.254440 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.254441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.254443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:53.254589 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.254590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.255339 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.255340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.255977 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.255979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.256051 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.256088 (+ 37us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.256119 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.256159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260201 03:08:53.256573 4323 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:53.256636 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.253265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.28.181.61:40901)
0201 03:08:53.254325 (+ 1060us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.254327 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.254334 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:53.254349 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.254350 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.254366 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.254421 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:53.254673 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.255267 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.256062 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.256473 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.256537 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.256559 (+ 22us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.256590 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044}
I20260201 03:08:53.256655 4323 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:53.256690 4323 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: stopping tablet replica
I20260201 03:08:53.256798 4277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 LEARNER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260201 03:08:53.257189 4229 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260201 03:08:53.257967 4277 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0867071355a643f59e50d5390fec86b1 (127.28.181.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.258850 4417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.258919 4417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:08:53.259024 4229 raft_consensus.cc:1064] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: attempting to promote NON_VOTER 72d8bc3b19e5459caa11176e27435bd1 to VOTER
I20260201 03:08:53.259024 4306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb9810ed2544bbb94c0263ac80809a9. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: NON_VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: true } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.259087 4306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:08:53.259173 4229 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } }
I20260201 03:08:53.259568 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.259440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48816)
0201 03:08:53.259489 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.259544 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.60:38849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260201 03:08:53.259610 4277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 LEARNER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260201 03:08:53.259802 4227 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260201 03:08:53.259991 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.259827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48820)
0201 03:08:53.259959 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.259977 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.60:38849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260201 03:08:53.260048 3646 consensus_peers.cc:597] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 -> Peer 0867071355a643f59e50d5390fec86b1 (127.28.181.60:38849): Couldn't send request to peer 0867071355a643f59e50d5390fec86b1. Status: Network error: Client connection negotiation failed: client connection to 127.28.181.60:38849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260201 03:08:53.260324 4229 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.260473 4277 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.261024 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.261060 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.261092 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:53.261106 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:53.261183 4306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.261232 4306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:08:53.268990 4323 master.cc:583] Master@127.28.181.60:38849 shutdown complete.
I20260201 03:08:53.269029 4323 master_runner.cc:305] Clearing existing system tablet
I20260201 03:08:53.269130 4323 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:53.269167 4323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:53.269687 4323 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:53.270228 4419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.270359 4323 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:53.270409 4323 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.270432 4323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
uuid: "0867071355a643f59e50d5390fec86b1"
format_stamp: "Formatted at 2026-02-01 03:08:53 on dist-test-slave-ck1n"
server_key: "47ad93410187cd026872bbb1436f2ee4"
server_key_iv: "cb08fb4c737f5e8b5196e01f955dc3da"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.270961 4323 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:53.272063 4323 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260201 03:08:53.272116 4323 log.cc:1199] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260201 03:08:53.272262 4323 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Deleting consensus metadata
I20260201 03:08:53.272358 4323 master_runner.cc:315] Copying system tablet from 127.28.181.62:46111
I20260201 03:08:53.272945 4323 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: tablet copy: Beginning tablet copy session from remote peer at address 127.28.181.62:46111
I20260201 03:08:53.277493 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.273170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35538)
0201 03:08:53.273256 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.273265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.273318 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.273424 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.273424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.273518 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:53.273640 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.273641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.274455 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.274455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.274812 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.274814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.274882 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.275267 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:53.275270 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:53.275465 (+ 195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:53.276825 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.276826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.276826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.276955 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.277122 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:53.277123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:53.277123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:53.277143 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:53.277305 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:53.277306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:53.277332 (+ 26us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.277376 (+ 44us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.277435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":58,"threads_started":1}
I20260201 03:08:53.277534 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.273221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.28.181.62:46111)
0201 03:08:53.273254 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.273256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.273307 (+ 51us) server_negotiation.cc:408] Connection header received
0201 03:08:53.273365 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.273366 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.273373 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.273423 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.273698 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.274378 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.274887 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.275037 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.275539 (+ 502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.275542 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.275542 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.275547 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.276744 (+ 1197us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.277029 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.277030 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.277030 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.277050 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.277176 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.277177 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.277177 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.277256 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.277310 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.277394 (+ 84us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.277459 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:53.277688 3680 tablet_copy_service.cc:140] P bcb9810ed2544bbb94c0263ac80809a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0867071355a643f59e50d5390fec86b1 ({username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:35538)
I20260201 03:08:53.277729 3680 tablet_copy_service.cc:161] P bcb9810ed2544bbb94c0263ac80809a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0867071355a643f59e50d5390fec86b1 at {username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:35538: session id = 0867071355a643f59e50d5390fec86b1-00000000000000000000000000000000
I20260201 03:08:53.278204 3680 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260201 03:08:53.278523 4323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260201 03:08:53.279425 4323 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: tablet copy: Starting download of 0 data blocks...
I20260201 03:08:53.279534 4323 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: tablet copy: Starting download of 1 WAL segments...
I20260201 03:08:53.280216 4323 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260201 03:08:53.280956 3680 tablet_copy_service.cc:342] P bcb9810ed2544bbb94c0263ac80809a9: Request end of tablet copy session 0867071355a643f59e50d5390fec86b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:35538
I20260201 03:08:53.281010 3680 tablet_copy_service.cc:434] P bcb9810ed2544bbb94c0263ac80809a9: ending tablet copy session 0867071355a643f59e50d5390fec86b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0867071355a643f59e50d5390fec86b1
W20260201 03:08:53.282094 4323 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260201 03:08:53.282677 4323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:53.283676 4429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:53.283703 4426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.283774 4323 server_base.cc:1060] running on GCE node
W20260201 03:08:53.285877 4427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:53.286021 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:53.286149 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:53.287258 4323 hybrid_clock.cc:648] HybridClock initialized: now 1769915333287249 us; error 36 us; skew 500 ppm
Feb 01 03:08:53 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915333, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:53.288314 4323 init.cc:377] Logged in from keytab as oryx/127.28.181.60@KRBTEST.COM (short username oryx)
I20260201 03:08:53.288537 4323 webserver.cc:492] Webserver started at http://127.28.181.60:38273/ using document root <none> and password file <none>
I20260201 03:08:53.288637 4323 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:53.288668 4323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:53.289162 4323 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.289579 4436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.289695 4323 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:53.289740 4323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
uuid: "0867071355a643f59e50d5390fec86b1"
format_stamp: "Formatted at 2026-02-01 03:08:53 on dist-test-slave-ck1n"
server_key: "47ad93410187cd026872bbb1436f2ee4"
server_key_iv: "cb08fb4c737f5e8b5196e01f955dc3da"
server_key_version: "encryptionkey@0"
I20260201 03:08:53.289800 4323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:53.300351 4323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:53.301518 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:53.304006 4323 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.60:38849
I20260201 03:08:53.304226 4488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.60:38849 every 8 connection(s)
I20260201 03:08:53.304327 4323 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/master-2/data/info.pb
I20260201 03:08:53.304935 4489 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:08:53.305315 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Bootstrap starting.
I20260201 03:08:53.307695 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:08:53.307894 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Bootstrap complete.
I20260201 03:08:53.308058 4489 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } }
I20260201 03:08:53.308120 4489 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0867071355a643f59e50d5390fec86b1, State: Initialized, Role: LEARNER
I20260201 03:08:53.308151 4489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } }
I20260201 03:08:53.308295 4489 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:08:53.308384 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.308512 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:08:53.309722 4504 catalog_manager.cc:1269] Loaded cluster ID: b4e384d9a33d4269a52223d002ce4478
I20260201 03:08:53.309811 4504 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: loading cluster ID for follower catalog manager: success
I20260201 03:08:53.310794 4504 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: acquiring CA information for follower catalog manager: success
I20260201 03:08:53.311087 4504 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260201 03:08:53.314507 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.308139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.28.181.60:38849)
0201 03:08:53.308293 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.308294 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.308302 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:53.308317 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.308318 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.308326 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.308997 (+ 671us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:53.309030 (+ 33us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.310355 (+ 1325us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.311047 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.311236 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.312174 (+ 938us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:53.312176 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:53.312176 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:53.312181 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:53.313560 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.313911 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.313912 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.313912 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.313928 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:53.314066 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:53.314066 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:53.314066 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:53.314120 (+ 54us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:53.314205 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.314298 (+ 93us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.314354 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
I20260201 03:08:53.316715 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.315267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35546)
0201 03:08:53.315301 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.315309 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.315338 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.315432 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.315432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.315433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:53.315559 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.315560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.316067 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.316068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.316533 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.316534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.316608 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.316610 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.316638 (+ 28us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.316662 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260201 03:08:53.317009 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.315317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.28.181.62:46111)
0201 03:08:53.315352 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.315354 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.315360 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:53.315378 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.315379 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.315386 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.315430 (+ 44us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:53.315623 (+ 193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.316001 (+ 378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.316606 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.316930 (+ 324us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.316948 (+ 18us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.316963 (+ 15us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.316978 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:53.319433 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.317931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48834)
0201 03:08:53.317975 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:53.317985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:53.318018 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:53.318180 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:53.318181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:53.318182 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:53.318288 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.318289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.318809 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.318810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:53.319259 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:53.319260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.319330 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.319333 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:53.319360 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:53.319386 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260201 03:08:53.319751 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:53.317998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.28.181.60:38849)
0201 03:08:53.318072 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:53.318073 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:53.318081 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:53.318100 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:53.318101 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:53.318112 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:53.318176 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:53.318351 (+ 175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.318742 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:53.319339 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:53.319654 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:53.319685 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:53.319700 (+ 15us) server_negotiation.cc:300] Negotiation successful
0201 03:08:53.319719 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260201 03:08:53.523234 3853 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
I20260201 03:08:53.664466 3989 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
I20260201 03:08:53.786226 4463 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260201 03:08:53.786381 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb9810ed2544bbb94c0263ac80809a9. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:53.786486 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:08:53.794322 4125 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
I20260201 03:08:54.292462 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: NON_VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: true } } }
I20260201 03:08:54.292598 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:54.292881 4227 raft_consensus.cc:1064] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: attempting to promote NON_VOTER 0867071355a643f59e50d5390fec86b1 to VOTER
I20260201 03:08:54.293030 4227 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:54.293454 4463 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 LEARNER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260201 03:08:54.293483 4277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Refusing update from remote peer bcb9810ed2544bbb94c0263ac80809a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260201 03:08:54.293663 4227 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260201 03:08:54.293843 4505 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260201 03:08:54.294562 4505 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0867071355a643f59e50d5390fec86b1 (127.28.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.294724 4277 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0867071355a643f59e50d5390fec86b1 (127.28.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295491 4505 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295563 4505 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:54.295522 4506 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295615 4505 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295727 4463 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0867071355a643f59e50d5390fec86b1 (127.28.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295783 4511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.295862 4511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:08:54.295645 4506 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:54.295663 4505 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:54.296541 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb9810ed2544bbb94c0263ac80809a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:54.296630 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:08:54.361281 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.359638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35550)
0201 03:08:54.359737 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.359747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.359791 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.359920 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.359921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.359922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.360070 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.360071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.360639 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.360640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.361138 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.361139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.361189 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.361192 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.361212 (+ 20us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.361235 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":58,"threads_started":1}
I20260201 03:08:54.361613 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.359702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.28.181.62:46111)
0201 03:08:54.359823 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.359825 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.359833 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:54.359849 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.359850 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.359857 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.359908 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.360141 (+ 233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.360578 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.361195 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.361528 (+ 333us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.361535 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.361550 (+ 15us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.361572 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":78,"threads_started":1}
I20260201 03:08:54.363602 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.362088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38454)
0201 03:08:54.362136 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.362144 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.362184 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.362345 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.362345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.362347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.362444 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.362445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.362979 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.362980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.363467 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.363468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.363518 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.363520 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.363542 (+ 22us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.363562 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260201 03:08:54.364001 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.362156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.28.181.61:40901)
0201 03:08:54.362246 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.362247 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.362254 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:54.362271 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.362271 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.362279 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.362332 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.362492 (+ 160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.362897 (+ 405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.363528 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.363921 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.363927 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.363943 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.363962 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":62,"threads_started":1}
I20260201 03:08:54.366063 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.364495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48836)
0201 03:08:54.364539 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.364552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.364576 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.364754 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.364755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.364756 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.364863 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.364863 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.365406 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.365407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.365895 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.365897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.365964 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.365966 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.365992 (+ 26us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.366017 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260201 03:08:54.366420 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.364559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.28.181.60:38849)
0201 03:08:54.364662 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.364663 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.364669 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:08:54.364683 (+ 14us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.364684 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.364690 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.364751 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.364929 (+ 178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.365337 (+ 408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.365964 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.366339 (+ 375us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.366344 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.366361 (+ 17us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.366380 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260201 03:08:54.368623 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.367273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:35562)
0201 03:08:54.367349 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.367357 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.367402 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.367482 (+ 80us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.367482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.367483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.367591 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.367592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.368052 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.368053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.368498 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.368498 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.368545 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.368548 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.368567 (+ 19us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.368585 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":55,"threads_started":1}
I20260201 03:08:54.368923 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.367302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.28.181.62:46111)
0201 03:08:54.367333 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.367335 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.367406 (+ 71us) server_negotiation.cc:408] Connection header received
0201 03:08:54.367435 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.367436 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.367441 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.367475 (+ 34us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.367631 (+ 156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.368000 (+ 369us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.368552 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.368855 (+ 303us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.368859 (+ 4us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.368873 (+ 14us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.368891 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260201 03:08:54.370975 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.369393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.0.0.1:38456)
0201 03:08:54.369434 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.369443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.369474 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.369596 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.369596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.369597 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.369713 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.369714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.370311 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.370312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.370772 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.370773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.370856 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.370858 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.370894 (+ 36us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.370925 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260201 03:08:54.371279 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.369445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.28.181.61:40901)
0201 03:08:54.369477 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.369479 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.369486 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:54.369520 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.369521 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.369528 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.369592 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.369788 (+ 196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.370239 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.370856 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.371211 (+ 355us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.371216 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.371232 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.371248 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:08:54.372995 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.371719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.0.0.1:48842)
0201 03:08:54.371760 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.371769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.371808 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.371912 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.371913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.371914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.372003 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.372003 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.372449 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.372449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.372877 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.372878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.372925 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.372927 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.372946 (+ 19us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.372964 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260201 03:08:54.373358 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.371790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.28.181.60:38849)
0201 03:08:54.371843 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.371844 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.371851 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:54.371870 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.371871 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.371876 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.371912 (+ 36us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.372045 (+ 133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.372398 (+ 353us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.372931 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.373289 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.373294 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.373309 (+ 15us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.373326 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:54.373957 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3724
I20260201 03:08:54.378374 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:46441
--local_ip_for_outbound_sockets=127.28.181.1
--tserver_master_addrs=127.28.181.62:46111
--webserver_port=36089
--webserver_interface=127.28.181.1
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:54.449335 4519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:54.449517 4519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:54.449545 4519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:54.450817 4519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:54.450855 4519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:54.450903 4519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:08:54.452219 4519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:46441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=36089
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:54.452414 4519 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:54.452571 4519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:54.454592 4524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:54.454599 4527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:54.454595 4525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:54.454761 4519 server_base.cc:1060] running on GCE node
I20260201 03:08:54.454931 4519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:54.455120 4519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:54.456240 4519 hybrid_clock.cc:648] HybridClock initialized: now 1769915334456229 us; error 24 us; skew 500 ppm
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:54.458287 4519 init.cc:377] Logged in from keytab as oryx/127.28.181.1@KRBTEST.COM (short username oryx)
I20260201 03:08:54.458742 4519 webserver.cc:492] Webserver started at http://127.28.181.1:36089/ using document root <none> and password file <none>
I20260201 03:08:54.458869 4519 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:54.458906 4519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:54.459820 4519 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:08:54.460225 4534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.460372 4519 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.460433 4519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "81843fd47754456a914971183b283442"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "ab666e3fe9db79f0d9675371d557b430"
server_key_iv: "b196b6e14b1497714e1c052efc1be40e"
server_key_version: "encryptionkey@0"
I20260201 03:08:54.460608 4519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:54.481060 4519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:54.482445 4519 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:54.482523 4519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:54.482652 4519 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:54.482838 4519 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:54.482862 4519 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.482878 4519 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:54.482889 4519 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.487672 4519 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:46441
I20260201 03:08:54.487754 4647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:46441 every 8 connection(s)
I20260201 03:08:54.487988 4519 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:08:54.492316 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 4519
I20260201 03:08:54.492478 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3860
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:08:54.497062 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:38717 (local address 127.28.181.62:46111)
0201 03:08:54.488655 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.488657 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.488993 (+ 336us) server_negotiation.cc:408] Connection header received
0201 03:08:54.489417 (+ 424us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.489419 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.489429 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.489476 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.490352 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491040 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.491673 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491852 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.494336 (+ 2484us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.494339 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.494340 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.494345 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.495956 (+ 1611us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496368 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496369 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496369 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496388 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496537 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496537 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496621 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.496657 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.496900 (+ 243us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.496980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:54.497071 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:50595 (local address 127.28.181.61:40901)
0201 03:08:54.488902 (+ 330us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.488904 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.488914 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:54.489316 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.489317 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.489328 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.489352 (+ 24us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.490210 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.490876 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.491574 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491755 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.494163 (+ 2408us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.494167 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.494167 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.494173 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.495933 (+ 1760us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496365 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496366 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496366 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496388 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496537 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496537 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496631 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.496665 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.496898 (+ 233us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.496980 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260201 03:08:54.497104 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:54221 (local address 127.28.181.60:38849)
0201 03:08:54.488870 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.488872 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.489046 (+ 174us) server_negotiation.cc:408] Connection header received
0201 03:08:54.489306 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.489307 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.489318 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.489364 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.490179 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.490627 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.491243 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491405 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.494136 (+ 2731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.494139 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.494140 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.494145 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.495786 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496285 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496286 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496286 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496308 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496478 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.496479 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.496479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.496573 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.496606 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.496894 (+ 288us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.496984 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260201 03:08:54.497458 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.28.181.1:50595)
0201 03:08:54.488681 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.488708 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.489219 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.489413 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.489414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.489818 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.490116 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.490118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.490989 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.490989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.491499 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.491501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491560 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.491897 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.491901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.494083 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.496016 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.496018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496293 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496450 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.496450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496479 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496701 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.496701 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.496765 (+ 64us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.497190 (+ 425us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.497323 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":55,"thread_start_us":72,"threads_started":1}
I20260201 03:08:54.497458 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:38717)
0201 03:08:54.488911 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.488919 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.489219 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.489491 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.489492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.489817 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.490277 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.490278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491123 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.491124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.491581 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.491582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491897 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.492215 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.492219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.494251 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.496021 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.496021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496303 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496450 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.496450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496479 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496697 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.496698 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.496765 (+ 67us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.497141 (+ 376us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.497269 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260201 03:08:54.497829 4648 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:54.497875 4650 heartbeater.cc:344] Connected to a master server at 127.28.181.61:40901
I20260201 03:08:54.497926 4648 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.497926 4650 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.498080 4650 heartbeater.cc:507] Master 127.28.181.61:40901 requested a full tablet report, sending...
I20260201 03:08:54.498095 4648 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:54.498384 3660 ts_manager.cc:194] Re-registered known tserver with Master: 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:54.498513 4267 ts_manager.cc:194] Registered new tserver with Master: 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:54.498811 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:38717
I20260201 03:08:54.498239 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.488792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.28.181.1:54221)
0201 03:08:54.488989 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.488998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.489234 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.489410 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.489414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.489819 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.490076 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.490079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.490706 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.490707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.491160 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.491162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.491284 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.491844 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.491864 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.494046 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.495863 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.495868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.495870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496213 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496362 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.496363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.496363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.496427 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.496640 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.496643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.496762 (+ 119us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.497195 (+ 433us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.497408 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":106,"thread_start_us":58,"threads_started":1}
I20260201 03:08:54.499738 4649 heartbeater.cc:344] Connected to a master server at 127.28.181.60:38849
I20260201 03:08:54.499786 4649 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.499841 4649 heartbeater.cc:507] Master 127.28.181.60:38849 requested a full tablet report, sending...
I20260201 03:08:54.500105 4453 ts_manager.cc:194] Registered new tserver with Master: 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:54.500123 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:34343
--local_ip_for_outbound_sockets=127.28.181.2
--tserver_master_addrs=127.28.181.62:46111
--webserver_port=34417
--webserver_interface=127.28.181.2
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:54.567795 4655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:54.567938 4655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:54.567957 4655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:54.569197 4655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:54.569232 4655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:54.569324 4655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:08:54.570767 4655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:34343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=34417
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:54.570935 4655 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:54.571054 4655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:54.573192 4663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:54.573179 4660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:54.573276 4655 server_base.cc:1060] running on GCE node
W20260201 03:08:54.573179 4661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:54.573462 4655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:54.573593 4655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:54.574702 4655 hybrid_clock.cc:648] HybridClock initialized: now 1769915334574695 us; error 20 us; skew 500 ppm
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:54.576529 4655 init.cc:377] Logged in from keytab as oryx/127.28.181.2@KRBTEST.COM (short username oryx)
I20260201 03:08:54.576936 4655 webserver.cc:492] Webserver started at http://127.28.181.2:34417/ using document root <none> and password file <none>
I20260201 03:08:54.577105 4655 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:54.577204 4655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:54.578284 4655 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:54.578725 4670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.578873 4655 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.578944 4655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "3b429af07ea346829b21aec63290f890"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "3297de39fe791d466c17421aba5b54ce"
server_key_iv: "71327650196524e32b2db520c4919fe3"
server_key_version: "encryptionkey@0"
I20260201 03:08:54.579154 4655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:54.588099 4655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:54.589268 4655 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:54.589385 4655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:54.589550 4655 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:54.589823 4655 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:54.589859 4655 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.589895 4655 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:54.589922 4655 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.594794 4655 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:34343
I20260201 03:08:54.594839 4783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:34343 every 8 connection(s)
I20260201 03:08:54.595084 4655 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:54.603351 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:44109 (local address 127.28.181.62:46111)
0201 03:08:54.595971 (+ 98us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.595973 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.596201 (+ 228us) server_negotiation.cc:408] Connection header received
0201 03:08:54.596276 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.596278 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.596287 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.596332 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.597029 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597665 (+ 636us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.598327 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598490 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.600204 (+ 1714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.600207 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.600208 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.600214 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.601747 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.602274 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.602275 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.602276 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.602301 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.602585 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.602586 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.602587 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.602689 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.602732 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.603121 (+ 389us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.603229 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260201 03:08:54.603386 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:44109)
0201 03:08:54.596144 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.596150 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.596215 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.596423 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.596426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.596711 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.596937 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.596939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597743 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.597743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.598252 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.598253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598352 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.598702 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.598736 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.600115 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.601919 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.601924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.601926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.602176 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.602373 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.602373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.602374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.602449 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.602781 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.602785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.602932 (+ 147us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.603061 (+ 129us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.603194 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":43,"threads_started":1}
I20260201 03:08:54.603659 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 4655
I20260201 03:08:54.603761 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:43431 (local address 127.28.181.60:38849)
0201 03:08:54.595963 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.595965 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.596091 (+ 126us) server_negotiation.cc:408] Connection header received
0201 03:08:54.596311 (+ 220us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.596312 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.596322 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.596350 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.597104 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597603 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.598336 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598494 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.601041 (+ 2547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.601045 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.601045 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.601054 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.602638 (+ 1584us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603112 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.603113 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.603114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.603135 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603281 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.603282 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.603283 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.603365 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.603402 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.603603 (+ 201us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.603677 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260201 03:08:54.603791 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3996
I20260201 03:08:54.603961 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:38117 (local address 127.28.181.61:40901)
0201 03:08:54.595749 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.595752 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.595878 (+ 126us) server_negotiation.cc:408] Connection header received
0201 03:08:54.596311 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.596312 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.596322 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.596350 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.597263 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597909 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.598422 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598842 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.601094 (+ 2252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.601097 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.601097 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.601102 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.602966 (+ 1864us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603313 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.603314 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.603314 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.603333 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603485 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.603485 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.603486 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.603566 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.603603 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.603779 (+ 176us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.603863 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:08:54.604250 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.28.181.2:38117)
0201 03:08:54.595781 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.595807 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.596213 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.596423 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.596426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.596749 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.597192 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.597193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597988 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.597989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.598342 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.598344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598877 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.599229 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.599233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.601023 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.603072 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.603073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.603241 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.603394 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.603395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.603421 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.603643 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.603644 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.603676 (+ 32us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.604045 (+ 369us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.604120 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260201 03:08:54.604892 4784 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:54.604892 4786 heartbeater.cc:344] Connected to a master server at 127.28.181.61:40901
I20260201 03:08:54.604995 4784 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.605007 4786 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.605170 4786 heartbeater.cc:507] Master 127.28.181.61:40901 requested a full tablet report, sending...
I20260201 03:08:54.605175 4784 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:54.605492 3660 ts_manager.cc:194] Re-registered known tserver with Master: 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
I20260201 03:08:54.605611 4267 ts_manager.cc:194] Registered new tserver with Master: 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
I20260201 03:08:54.605947 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:44109
I20260201 03:08:54.606453 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.595604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.28.181.2:43431)
0201 03:08:54.596015 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.596023 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.596213 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.596423 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.596426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.596740 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.597009 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.597011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.597690 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.597690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.598252 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.598253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.598540 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.598785 (+ 245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.598788 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.600959 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.602776 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.602778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.602778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.603017 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.603198 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.603198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.603199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.603224 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.603446 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.603447 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.603496 (+ 49us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.606218 (+ 2722us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.606321 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":65,"threads_started":1}
I20260201 03:08:54.609300 4785 heartbeater.cc:344] Connected to a master server at 127.28.181.60:38849
I20260201 03:08:54.609344 4785 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.609375 4785 heartbeater.cc:507] Master 127.28.181.60:38849 requested a full tablet report, sending...
I20260201 03:08:54.609635 4453 ts_manager.cc:194] Registered new tserver with Master: 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
I20260201 03:08:54.609710 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:41021
--local_ip_for_outbound_sockets=127.28.181.3
--tserver_master_addrs=127.28.181.62:46111
--webserver_port=34543
--webserver_interface=127.28.181.3
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849 with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:08:54.675551 4791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:08:54.675769 4791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:08:54.675801 4791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:08:54.677356 4791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:08:54.677414 4791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:08:54.677451 4791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:08:54.678761 4791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:41021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=34543
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:46111,127.28.181.61:40901,127.28.181.60:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:08:54.678977 4791 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:08:54.679174 4791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:08:54.681389 4797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:08:54.681404 4799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:54.681506 4791 server_base.cc:1060] running on GCE node
W20260201 03:08:54.681396 4796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:08:54.681728 4791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:08:54.681967 4791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:08:54.683082 4791 hybrid_clock.cc:648] HybridClock initialized: now 1769915334683081 us; error 32 us; skew 500 ppm
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:08:54.684973 4791 init.cc:377] Logged in from keytab as oryx/127.28.181.3@KRBTEST.COM (short username oryx)
I20260201 03:08:54.685384 4791 webserver.cc:492] Webserver started at http://127.28.181.3:34543/ using document root <none> and password file <none>
I20260201 03:08:54.685540 4791 fs_manager.cc:362] Metadata directory not provided
I20260201 03:08:54.685597 4791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:08:54.686714 4791 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.687237 4806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.687383 4791 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.687453 4791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "d1801f0dfc7c44da984786c68e85c06f"
format_stamp: "Formatted at 2026-02-01 03:08:52 on dist-test-slave-ck1n"
server_key: "4c034fcf90b85397b0f645a1daf162f9"
server_key_iv: "f78be5cc7da415fe6c6cd22fecad19d3"
server_key_version: "encryptionkey@0"
I20260201 03:08:54.687659 4791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:08:54.694921 4791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:08:54.696475 4791 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:08:54.696590 4791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:08:54.696760 4791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:08:54.696983 4791 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:08:54.697028 4791 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.697060 4791 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:08:54.697082 4791 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:54.701704 4791 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:41021
I20260201 03:08:54.701756 4919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:41021 every 8 connection(s)
I20260201 03:08:54.702000 4791 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:08:54.703104 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 4791
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.60@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
Feb 01 03:08:54 dist-test-slave-ck1n krb5kdc[3612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915334, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.61@KRBTEST.COM
I20260201 03:08:54.712007 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:51637)
0201 03:08:54.702849 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.702858 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.703456 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.703637 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.703638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.703830 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.704185 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.704188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.704886 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.704886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.705266 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.705268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705393 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.706116 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.706120 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.708996 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.710780 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.710785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.710787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.711026 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.711193 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.711194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.711194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.711258 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.711493 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.711497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.711609 (+ 112us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.711710 (+ 101us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.711830 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":49,"threads_started":1}
I20260201 03:08:54.712286 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:51637 (local address 127.28.181.62:46111)
0201 03:08:54.702842 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.702844 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.702920 (+ 76us) server_negotiation.cc:408] Connection header received
0201 03:08:54.703528 (+ 608us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.703529 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.703538 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.703585 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.704284 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.704800 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.705344 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705500 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.709199 (+ 3699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.709202 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.709202 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.709207 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.710700 (+ 1493us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.711106 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.711107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.711107 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.711129 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.711323 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.711324 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.711325 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.711419 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.711471 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.712122 (+ 651us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.712193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:08:54.712597 4920 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46111
I20260201 03:08:54.712670 4920 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.712855 4920 heartbeater.cc:507] Master 127.28.181.62:46111 requested a full tablet report, sending...
I20260201 03:08:54.713142 3660 ts_manager.cc:194] Re-registered known tserver with Master: d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021)
I20260201 03:08:54.713449 3660 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:51637
I20260201 03:08:54.714172 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.60:38849 (local address 127.28.181.3:53225)
0201 03:08:54.702599 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.702622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.703188 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.703368 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.703368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.703831 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.704185 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.704188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.704980 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.704981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.705495 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.705496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705583 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.706084 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.706087 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.708723 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.712835 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.712836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.712836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.713009 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.713693 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.713694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.713694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.713720 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.713954 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.713954 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.713982 (+ 28us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.714036 (+ 54us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.714118 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":58,"threads_started":1}
I20260201 03:08:54.714453 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:53225 (local address 127.28.181.60:38849)
0201 03:08:54.702578 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.702580 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.702679 (+ 99us) server_negotiation.cc:408] Connection header received
0201 03:08:54.703270 (+ 591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.703271 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.703281 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.703307 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.704273 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.704901 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.705861 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.706021 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.708794 (+ 2773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.708798 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.708798 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.708803 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.712769 (+ 3966us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.713608 (+ 839us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.713609 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.713609 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.713633 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.713773 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.713774 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.713774 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.713870 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.713931 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.714302 (+ 371us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.714377 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260201 03:08:54.714722 4921 heartbeater.cc:344] Connected to a master server at 127.28.181.60:38849
I20260201 03:08:54.714784 4921 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.714850 4921 heartbeater.cc:507] Master 127.28.181.60:38849 requested a full tablet report, sending...
I20260201 03:08:54.715102 4453 ts_manager.cc:194] Registered new tserver with Master: d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021)
I20260201 03:08:54.715397 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.704915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.28.181.61:40901)
0201 03:08:54.705908 (+ 993us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.705910 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.706213 (+ 303us) server_negotiation.cc:408] Connection header received
0201 03:08:54.709108 (+ 2895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.709109 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.709118 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.709169 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.710008 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.710700 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.711470 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.711623 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.712214 (+ 591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.712217 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.712218 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.712223 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.713874 (+ 1651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.714318 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.714319 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.714319 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.714339 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.714665 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.714666 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.714666 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.714755 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.714895 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.715216 (+ 321us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.715291 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":66,"threads_started":1}
I20260201 03:08:54.715574 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:58547 (local address 127.28.181.61:40901)
0201 03:08:54.702871 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.702873 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.702882 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:08:54.703259 (+ 377us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.703260 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.703271 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.703301 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.704368 (+ 1067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.704973 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.705602 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705807 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.709019 (+ 3212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.709022 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.709023 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.709028 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.712118 (+ 3090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.714885 (+ 2767us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.714886 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.714887 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.714908 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.715053 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.715054 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.715054 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.715151 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.715204 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.715402 (+ 198us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.715482 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:08:54.715719 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.702492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.61:40901 (local address 127.28.181.3:58547)
0201 03:08:54.702629 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.702638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.703181 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.703361 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.703364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.703822 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:08:54.704279 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.704280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705046 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.705046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.705495 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.705496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.705582 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.706057 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:08:54.706077 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:08:54.708726 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:08:54.712859 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.712859 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.712860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.714792 (+ 1932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.714960 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:08:54.714961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:08:54.714961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:08:54.714987 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:08:54.715262 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:08:54.715263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:08:54.715296 (+ 33us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.715361 (+ 65us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.715599 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260201 03:08:54.716046 4922 heartbeater.cc:344] Connected to a master server at 127.28.181.61:40901
I20260201 03:08:54.716091 4922 heartbeater.cc:461] Registering TS with master...
I20260201 03:08:54.716132 4922 heartbeater.cc:507] Master 127.28.181.61:40901 requested a full tablet report, sending...
I20260201 03:08:54.716404 4267 ts_manager.cc:194] Registered new tserver with Master: d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021)
I20260201 03:08:54.717164 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.706378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.28.181.60:38849)
0201 03:08:54.706515 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.706517 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.706943 (+ 426us) server_negotiation.cc:408] Connection header received
0201 03:08:54.706970 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.706971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.706981 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.707041 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.707725 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.708392 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.709733 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.709911 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.710698 (+ 787us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.710700 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.710701 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.710706 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.714156 (+ 3450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.716645 (+ 2489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.716646 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.716646 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.716664 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.716776 (+ 112us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.716777 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.716777 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.716858 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.716932 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.717015 (+ 83us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.717085 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260201 03:08:54.717646 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.705924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.28.181.62:46111)
0201 03:08:54.706699 (+ 775us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.706701 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.706711 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:54.706728 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.706729 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.706739 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.706797 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.707548 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.708176 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.708840 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.714283 (+ 5443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.714320 (+ 37us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.714323 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.714323 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.714326 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.716270 (+ 1944us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.716797 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.716798 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.716798 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.716823 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.716961 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.716962 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.716963 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.717034 (+ 71us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.717125 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.717516 (+ 391us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.717597 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":67,"threads_started":1}
I20260201 03:08:54.718736 3660 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:54.719908 3660 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:54.725461 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:41021 (local address 127.0.0.1:35428)
0201 03:08:54.723310 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.723319 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.723366 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.723818 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.723819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.723820 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.723974 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.723975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.724679 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.724680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.725294 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.725295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.725350 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.725353 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.725380 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.725407 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":55,"threads_started":1}
I20260201 03:08:54.725692 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:34343 (local address 127.0.0.1:43156)
0201 03:08:54.723191 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.723199 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.723245 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.723586 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.723587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.723589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.723760 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.723761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.724848 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.724849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.725504 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.725506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.725562 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.725564 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.725585 (+ 21us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.725634 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260201 03:08:54.726251 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.28.181.3:41021)
0201 03:08:54.723681 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.723690 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.723698 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:54.723723 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.723724 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.723737 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.723797 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.724035 (+ 238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.724607 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.725651 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.726150 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.726175 (+ 25us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.726196 (+ 21us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.726215 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":58,"threads_started":1}
I20260201 03:08:54.726257 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.28.181.2:34343)
0201 03:08:54.723374 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.723433 (+ 59us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.723443 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:54.723477 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.723479 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.723494 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.723581 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.723850 (+ 269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.724766 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.725843 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.726147 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.726174 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.726192 (+ 18us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.726213 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260201 03:08:54.726712 4718 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 (DEFAULT_TABLE table=test-table [id=7f60a96d46e84c6c82f2cb6669798f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:54.726881 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46441 (local address 127.0.0.1:59358)
0201 03:08:54.723663 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.723672 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.723722 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.723880 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.723880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.723890 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.724062 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.724063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.725087 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.725088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.726685 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.726687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.726754 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.726756 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.726784 (+ 28us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.726816 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":68,"threads_started":1}
I20260201 03:08:54.726830 4854 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 (DEFAULT_TABLE table=test-table [id=7f60a96d46e84c6c82f2cb6669798f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:54.726966 4718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe03e728b584c6ab1935bc2c7b2b983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:54.727056 4854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe03e728b584c6ab1935bc2c7b2b983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:54.727365 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.723393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.28.181.1:46441)
0201 03:08:54.723541 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.723552 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.723774 (+ 222us) server_negotiation.cc:408] Connection header received
0201 03:08:54.723809 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.723810 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.723823 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.723879 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.724133 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.725017 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.726766 (+ 1749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.727204 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.727237 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.727262 (+ 25us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.727301 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260201 03:08:54.727800 4582 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 (DEFAULT_TABLE table=test-table [id=7f60a96d46e84c6c82f2cb6669798f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:54.728024 4582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe03e728b584c6ab1935bc2c7b2b983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:54.728941 4949 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Bootstrap starting.
I20260201 03:08:54.729544 4950 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Bootstrap starting.
I20260201 03:08:54.729594 4949 tablet_bootstrap.cc:654] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:54.729853 4949 log.cc:826] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:54.730362 4950 tablet_bootstrap.cc:654] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:54.730371 4949 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: No bootstrap required, opened a new log
I20260201 03:08:54.730434 4949 ts_tablet_manager.cc:1403] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260201 03:08:54.730505 4952 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Bootstrap starting.
I20260201 03:08:54.730739 4950 log.cc:826] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:54.731271 4950 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: No bootstrap required, opened a new log
I20260201 03:08:54.731345 4950 ts_tablet_manager.cc:1403] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260201 03:08:54.731424 4952 tablet_bootstrap.cc:654] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:54.731665 4952 log.cc:826] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Log is configured to *not* fsync() on all Append() calls
I20260201 03:08:54.732065 4949 raft_consensus.cc:359] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.732183 4949 raft_consensus.cc:385] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:54.732211 4949 raft_consensus.cc:740] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b429af07ea346829b21aec63290f890, State: Initialized, Role: FOLLOWER
I20260201 03:08:54.732327 4952 tablet_bootstrap.cc:492] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: No bootstrap required, opened a new log
I20260201 03:08:54.732327 4949 consensus_queue.cc:260] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.732379 4952 ts_tablet_manager.cc:1403] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:54.732523 4949 ts_tablet_manager.cc:1434] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:54.732569 4784 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
I20260201 03:08:54.732676 4950 raft_consensus.cc:359] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.732904 4950 raft_consensus.cc:385] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:54.732941 4950 raft_consensus.cc:740] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1801f0dfc7c44da984786c68e85c06f, State: Initialized, Role: FOLLOWER
I20260201 03:08:54.733117 4950 consensus_queue.cc:260] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.733374 4920 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
I20260201 03:08:54.733315 4950 ts_tablet_manager.cc:1434] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:08:54.734650 4952 raft_consensus.cc:359] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.734735 4952 raft_consensus.cc:385] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:54.734750 4952 raft_consensus.cc:740] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81843fd47754456a914971183b283442, State: Initialized, Role: FOLLOWER
I20260201 03:08:54.734833 4952 consensus_queue.cc:260] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.734977 4952 ts_tablet_manager.cc:1434] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:08:54.735087 4648 heartbeater.cc:499] Master 127.28.181.62:46111 was elected leader, sending a full tablet report...
W20260201 03:08:54.738896 4651 tablet.cc:2378] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:54.817467 4955 raft_consensus.cc:493] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:54.817548 4955 raft_consensus.cc:515] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.817864 4955 leader_election.cc:290] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021), 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:54.819679 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.817996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:41021 (local address 127.28.181.2:56933)
0201 03:08:54.818074 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.818084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.818132 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.818260 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.818260 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.818262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.818428 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.818428 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.818969 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.818970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.819446 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.819447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.819521 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.819525 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.819552 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.819586 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260201 03:08:54.819679 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.818019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46441 (local address 127.28.181.2:56099)
0201 03:08:54.818095 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:54.818104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:54.818157 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:54.818310 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:54.818311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:54.818312 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:54.818468 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.818469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.819000 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.819001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:54.819467 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:54.819468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.819544 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.819547 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:54.819574 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:54.819605 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260201 03:08:54.820077 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.818054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:56933 (local address 127.28.181.3:41021)
0201 03:08:54.818090 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.818092 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.818128 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:08:54.818184 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.818186 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.818193 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.818257 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.818502 (+ 245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.818898 (+ 396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.819519 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.819954 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.819962 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.819984 (+ 22us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.820018 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:08:54.820104 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.818066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:56099 (local address 127.28.181.1:46441)
0201 03:08:54.818103 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.818105 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.818141 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:08:54.818227 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.818228 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.818237 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.818306 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:54.818534 (+ 228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.818928 (+ 394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.819544 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.819992 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.820000 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.820022 (+ 22us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.820050 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:54.820334 4874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe03e728b584c6ab1935bc2c7b2b983" candidate_uuid: "3b429af07ea346829b21aec63290f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1801f0dfc7c44da984786c68e85c06f" is_pre_election: true
I20260201 03:08:54.820359 4602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe03e728b584c6ab1935bc2c7b2b983" candidate_uuid: "3b429af07ea346829b21aec63290f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81843fd47754456a914971183b283442" is_pre_election: true
I20260201 03:08:54.820432 4874 raft_consensus.cc:2468] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b429af07ea346829b21aec63290f890 in term 0.
I20260201 03:08:54.820484 4602 raft_consensus.cc:2468] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b429af07ea346829b21aec63290f890 in term 0.
I20260201 03:08:54.820644 4671 leader_election.cc:304] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b429af07ea346829b21aec63290f890, d1801f0dfc7c44da984786c68e85c06f; no voters:
I20260201 03:08:54.820762 4955 raft_consensus.cc:2804] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:54.820833 4955 raft_consensus.cc:493] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:54.820866 4955 raft_consensus.cc:3060] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:54.821439 4955 raft_consensus.cc:515] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.821578 4955 leader_election.cc:290] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [CANDIDATE]: Term 1 election: Requested vote from peers d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021), 81843fd47754456a914971183b283442 (127.28.181.1:46441)
I20260201 03:08:54.821713 4874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe03e728b584c6ab1935bc2c7b2b983" candidate_uuid: "3b429af07ea346829b21aec63290f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1801f0dfc7c44da984786c68e85c06f"
I20260201 03:08:54.821717 4602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe03e728b584c6ab1935bc2c7b2b983" candidate_uuid: "3b429af07ea346829b21aec63290f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81843fd47754456a914971183b283442"
I20260201 03:08:54.821797 4874 raft_consensus.cc:3060] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:54.821763 4602 raft_consensus.cc:3060] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:54.822347 4874 raft_consensus.cc:2468] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b429af07ea346829b21aec63290f890 in term 1.
I20260201 03:08:54.822387 4602 raft_consensus.cc:2468] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b429af07ea346829b21aec63290f890 in term 1.
I20260201 03:08:54.822507 4671 leader_election.cc:304] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b429af07ea346829b21aec63290f890, d1801f0dfc7c44da984786c68e85c06f; no voters:
I20260201 03:08:54.822592 4955 raft_consensus.cc:2804] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:54.822723 4955 raft_consensus.cc:697] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 1 LEADER]: Becoming Leader. State: Replica: 3b429af07ea346829b21aec63290f890, State: Running, Role: LEADER
I20260201 03:08:54.822825 4955 consensus_queue.cc:237] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } }
I20260201 03:08:54.823511 3659 catalog_manager.cc:5654] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b429af07ea346829b21aec63290f890 (127.28.181.2). New cstate: current_term: 1 leader_uuid: "3b429af07ea346829b21aec63290f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260201 03:08:54.839792 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.838159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.28.181.2:34343)
0201 03:08:54.838211 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.838213 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.838220 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:54.838271 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.838275 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.838286 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.838343 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:54.838530 (+ 187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.838898 (+ 368us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.839399 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.839523 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.839548 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:54.839610 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:54.839684 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.839717 (+ 33us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.839735 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:08:54.841008 4602 raft_consensus.cc:1275] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Refusing update from remote peer 3b429af07ea346829b21aec63290f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:54.841022 4874 raft_consensus.cc:1275] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Refusing update from remote peer 3b429af07ea346829b21aec63290f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:54.841260 4955 consensus_queue.cc:1048] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:54.841375 4958 consensus_queue.cc:1048] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:54.843698 4963 mvcc.cc:204] Tried to move back new op lower bound from 7249573211506708480 to 7249573211434795008. Current Snapshot: MvccSnapshot[applied={T|T < 7249573211506708480}]
I20260201 03:08:54.844241 4962 mvcc.cc:204] Tried to move back new op lower bound from 7249573211506708480 to 7249573211434795008. Current Snapshot: MvccSnapshot[applied={T|T < 7249573211506708480 or (T in {7249573211506708480})}]
I20260201 03:08:54.845575 3659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:54.845649 3659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:54.846868 3659 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bcb9810ed2544bbb94c0263ac80809a9: Sending DeleteTablet for 3 replicas of tablet 0fe03e728b584c6ab1935bc2c7b2b983
I20260201 03:08:54.847187 4718 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:54 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:43156
I20260201 03:08:54.847241 4854 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:54 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260201 03:08:54.847412 4972 tablet_replica.cc:333] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: stopping tablet replica
I20260201 03:08:54.847510 4972 raft_consensus.cc:2243] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:54.847625 4972 raft_consensus.cc:2272] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:54.847965 4582 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe03e728b584c6ab1935bc2c7b2b983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:54 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:59358
I20260201 03:08:54.848245 4972 ts_tablet_manager.cc:1916] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:54.848248 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 3628
I20260201 03:08:54.849905 4972 ts_tablet_manager.cc:1929] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:54.850016 4972 log.cc:1199] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/0fe03e728b584c6ab1935bc2c7b2b983
I20260201 03:08:54.850306 4972 ts_tablet_manager.cc:1950] T 0fe03e728b584c6ab1935bc2c7b2b983 P d1801f0dfc7c44da984786c68e85c06f: Deleting consensus metadata
I20260201 03:08:54.852375 4973 tablet_replica.cc:333] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: stopping tablet replica
I20260201 03:08:54.852451 4973 raft_consensus.cc:2243] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:54.852528 4973 raft_consensus.cc:2272] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:54.852661 4971 tablet_replica.cc:333] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: stopping tablet replica
I20260201 03:08:54.852725 4971 raft_consensus.cc:2243] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:54.852818 4971 raft_consensus.cc:2272] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:54.853032 4973 ts_tablet_manager.cc:1916] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:54.853138 4971 ts_tablet_manager.cc:1916] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260201 03:08:54.853686 4810 connection.cc:537] server connection from 127.0.0.1:35428 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260201 03:08:54.854446 4973 ts_tablet_manager.cc:1929] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:54.854558 4973 log.cc:1199] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/0fe03e728b584c6ab1935bc2c7b2b983
I20260201 03:08:54.854856 4973 ts_tablet_manager.cc:1950] T 0fe03e728b584c6ab1935bc2c7b2b983 P 81843fd47754456a914971183b283442: Deleting consensus metadata
I20260201 03:08:54.855006 4971 ts_tablet_manager.cc:1929] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:54.855064 4971 log.cc:1199] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/0fe03e728b584c6ab1935bc2c7b2b983
W20260201 03:08:54.855245 4536 connection.cc:441] server connection from 127.0.0.1:59358 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59358 (request call id 1) could send its response
I20260201 03:08:54.855427 4971 ts_tablet_manager.cc:1950] T 0fe03e728b584c6ab1935bc2c7b2b983 P 3b429af07ea346829b21aec63290f890: Deleting consensus metadata
W20260201 03:08:54.855674 4673 connection.cc:441] server connection from 127.0.0.1:43156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43156 (request call id 1) could send its response
I20260201 03:08:54.861868 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.856081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.28.181.60:38849)
0201 03:08:54.856115 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.856117 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.856200 (+ 83us) server_negotiation.cc:408] Connection header received
0201 03:08:54.856258 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.856259 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.856269 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.856317 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.856864 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.857538 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.858146 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.858323 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.859356 (+ 1033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.859359 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.859359 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.859365 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.860894 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.861213 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.861214 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.861214 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.861233 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.861367 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.861367 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.861368 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.861460 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.861542 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.861659 (+ 117us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.861715 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:08:54.862373 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:54.855685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.28.181.61:40901)
0201 03:08:54.855753 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:54.855755 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:54.855792 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:08:54.855845 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:54.855846 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:54.855855 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:54.855905 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:08:54.856960 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.857586 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:54.858336 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:54.858498 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:54.859590 (+ 1092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:08:54.859593 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:08:54.859594 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:08:54.859599 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:08:54.861137 (+ 1538us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.861442 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.861443 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.861443 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.861460 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:08:54.861918 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:08:54.861919 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:08:54.861919 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:08:54.862013 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:08:54.862083 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:54.862218 (+ 135us) server_negotiation.cc:300] Negotiation successful
0201 03:08:54.862294 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:08:55.844816 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:55.844613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:47961)
0201 03:08:55.844713 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:55.844763 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260201 03:08:55.845037 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:55.844808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:32869)
0201 03:08:55.844913 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:55.844997 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
W20260201 03:08:55.845060 4920 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:46111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
W20260201 03:08:55.845206 4784 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:46111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:55.846191 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:55.845969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:56643)
0201 03:08:55.846092 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:55.846153 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":42,"threads_started":1}
W20260201 03:08:55.846349 4648 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:46111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.347729 4984 raft_consensus.cc:493] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcb9810ed2544bbb94c0263ac80809a9)
I20260201 03:08:56.347854 4985 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcb9810ed2544bbb94c0263ac80809a9)
I20260201 03:08:56.347868 4984 raft_consensus.cc:515] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:56.347970 4985 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:56.348119 4984 leader_election.cc:290] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 0867071355a643f59e50d5390fec86b1 (127.28.181.60:38849)
I20260201 03:08:56.348197 4985 leader_election.cc:290] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61:40901)
I20260201 03:08:56.348385 4277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0867071355a643f59e50d5390fec86b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72d8bc3b19e5459caa11176e27435bd1" is_pre_election: true
I20260201 03:08:56.348398 4463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8bc3b19e5459caa11176e27435bd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0867071355a643f59e50d5390fec86b1" is_pre_election: true
I20260201 03:08:56.348465 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.348214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36028)
0201 03:08:56.348394 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.348430 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":50,"threads_started":1}
I20260201 03:08:56.348477 4463 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d8bc3b19e5459caa11176e27435bd1 in term 1.
I20260201 03:08:56.348480 4277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0867071355a643f59e50d5390fec86b1 in term 1.
I20260201 03:08:56.348553 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.348394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36032)
0201 03:08:56.348487 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.348519 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":40,"threads_started":1}
W20260201 03:08:56.348574 4253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
W20260201 03:08:56.348610 4439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.348618 4440 leader_election.cc:304] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0867071355a643f59e50d5390fec86b1, 72d8bc3b19e5459caa11176e27435bd1; no voters:
I20260201 03:08:56.348729 4985 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260201 03:08:56.348775 4985 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcb9810ed2544bbb94c0263ac80809a9)
I20260201 03:08:56.348801 4985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 1 FOLLOWER]: Advancing to term 2
I20260201 03:08:56.348850 4253 leader_election.cc:304] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0867071355a643f59e50d5390fec86b1, 72d8bc3b19e5459caa11176e27435bd1; no voters:
I20260201 03:08:56.348960 4984 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260201 03:08:56.349011 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.348867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36048)
0201 03:08:56.348954 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.348982 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":39,"threads_started":1}
I20260201 03:08:56.349041 4984 raft_consensus.cc:493] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcb9810ed2544bbb94c0263ac80809a9)
I20260201 03:08:56.349071 4984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 1 FOLLOWER]: Advancing to term 2
W20260201 03:08:56.349076 4439 leader_election.cc:336] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.349006 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.348808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36042)
0201 03:08:56.348945 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.348969 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260201 03:08:56.349201 4253 leader_election.cc:336] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.349579 4985 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:56.349704 4985 leader_election.cc:290] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 election: Requested vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61:40901)
I20260201 03:08:56.349709 4984 raft_consensus.cc:515] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:56.349845 4984 leader_election.cc:290] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 election: Requested vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 0867071355a643f59e50d5390fec86b1 (127.28.181.60:38849)
I20260201 03:08:56.349884 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.349807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36062)
0201 03:08:56.349839 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.349858 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260201 03:08:56.349886 4277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0867071355a643f59e50d5390fec86b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72d8bc3b19e5459caa11176e27435bd1"
I20260201 03:08:56.349939 4277 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0867071355a643f59e50d5390fec86b1 in current term 2: Already voted for candidate 72d8bc3b19e5459caa11176e27435bd1 in this term.
I20260201 03:08:56.350031 4463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8bc3b19e5459caa11176e27435bd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0867071355a643f59e50d5390fec86b1"
I20260201 03:08:56.350107 4463 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 72d8bc3b19e5459caa11176e27435bd1 in current term 2: Already voted for candidate 0867071355a643f59e50d5390fec86b1 in this term.
I20260201 03:08:56.350183 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.350110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36082)
0201 03:08:56.350141 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.350159 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260201 03:08:56.350243 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.349916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36066)
0201 03:08:56.350190 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.350218 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
W20260201 03:08:56.350256 4439 leader_election.cc:336] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.350279 4439 leader_election.cc:304] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0867071355a643f59e50d5390fec86b1; no voters: 72d8bc3b19e5459caa11176e27435bd1, bcb9810ed2544bbb94c0263ac80809a9
I20260201 03:08:56.350364 4985 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260201 03:08:56.350489 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.350417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:36096)
0201 03:08:56.350450 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.350471 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260201 03:08:56.350554 4253 leader_election.cc:336] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:56.350592 4253 leader_election.cc:304] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8bc3b19e5459caa11176e27435bd1; no voters: 0867071355a643f59e50d5390fec86b1, bcb9810ed2544bbb94c0263ac80809a9
I20260201 03:08:56.350646 4984 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260201 03:08:56.845525 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.845324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:42767)
0201 03:08:56.845451 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.845479 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260201 03:08:56.845703 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.845512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:54513)
0201 03:08:56.845615 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.845646 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260201 03:08:56.846728 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:56.846577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:46271)
0201 03:08:56.846671 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:56.846698 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":54,"threads_started":1}
I20260201 03:08:57.846174 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:57.845957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:46093)
0201 03:08:57.846079 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:57.846112 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260201 03:08:57.846280 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:57.846070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:37799)
0201 03:08:57.846181 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:57.846214 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":45,"threads_started":1}
I20260201 03:08:57.847234 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:57.847083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:42173)
0201 03:08:57.847168 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:57.847193 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":54,"threads_started":1}
I20260201 03:08:58.506299 4997 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:58.506384 4997 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:58.506546 4997 leader_election.cc:290] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61:40901)
I20260201 03:08:58.506721 4277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0867071355a643f59e50d5390fec86b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72d8bc3b19e5459caa11176e27435bd1" is_pre_election: true
I20260201 03:08:58.506805 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.506638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45030)
0201 03:08:58.506746 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.506770 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260201 03:08:58.506814 4277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0867071355a643f59e50d5390fec86b1 in term 2.
I20260201 03:08:58.506937 4440 leader_election.cc:304] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0867071355a643f59e50d5390fec86b1, 72d8bc3b19e5459caa11176e27435bd1; no voters:
I20260201 03:08:58.507043 4997 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260201 03:08:58.507082 4997 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:58.507102 4997 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 2 FOLLOWER]: Advancing to term 3
I20260201 03:08:58.507162 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45036)
0201 03:08:58.507117 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.507138 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260201 03:08:58.507231 4439 leader_election.cc:336] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:58.507755 4997 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:58.507902 4997 leader_election.cc:290] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 election: Requested vote from peers bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111), 72d8bc3b19e5459caa11176e27435bd1 (127.28.181.61:40901)
I20260201 03:08:58.508040 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.507958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45048)
0201 03:08:58.507991 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.508016 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260201 03:08:58.508033 4277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0867071355a643f59e50d5390fec86b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72d8bc3b19e5459caa11176e27435bd1"
I20260201 03:08:58.508121 4277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 2 FOLLOWER]: Advancing to term 3
I20260201 03:08:58.508378 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.508303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45052)
0201 03:08:58.508341 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.508362 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260201 03:08:58.508455 4439 leader_election.cc:336] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
I20260201 03:08:58.508818 4277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0867071355a643f59e50d5390fec86b1 in term 3.
I20260201 03:08:58.508986 4440 leader_election.cc:304] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0867071355a643f59e50d5390fec86b1, 72d8bc3b19e5459caa11176e27435bd1; no voters: bcb9810ed2544bbb94c0263ac80809a9
I20260201 03:08:58.509094 4997 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260201 03:08:58.509219 4997 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [term 3 LEADER]: Becoming Leader. State: Replica: 0867071355a643f59e50d5390fec86b1, State: Running, Role: LEADER
I20260201 03:08:58.509281 4997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } }
I20260201 03:08:58.509565 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0867071355a643f59e50d5390fec86b1. Latest consensus state: current_term: 3 leader_uuid: "0867071355a643f59e50d5390fec86b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:58.509650 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:58.509825 5002 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:08:58.510197 5002 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:08:58.510252 5002 catalog_manager.cc:1269] Loaded cluster ID: b4e384d9a33d4269a52223d002ce4478
I20260201 03:08:58.510298 5002 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:08:58.510468 5002 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:08:58.510542 5002 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Loaded TSK: 0
I20260201 03:08:58.510689 5002 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:08:58.577438 4453 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:08:58.578687 4453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:08:58.579558 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.579462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45070)
0201 03:08:58.579505 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.579536 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260201 03:08:58.579561 4277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [term 3 FOLLOWER]: Refusing update from remote peer 0867071355a643f59e50d5390fec86b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260201 03:08:58.579793 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.579741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.0.0.1:45080)
0201 03:08:58.579762 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.579780 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260201 03:08:58.579752 5000 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260201 03:08:58.579922 4439 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 -> Peer bcb9810ed2544bbb94c0263ac80809a9 (127.28.181.62:46111): Couldn't send request to peer bcb9810ed2544bbb94c0263ac80809a9. Status: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260201 03:08:58.580331 5004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0867071355a643f59e50d5390fec86b1. Latest consensus state: current_term: 3 leader_uuid: "0867071355a643f59e50d5390fec86b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:58.580417 5004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:08:58.580765 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0867071355a643f59e50d5390fec86b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:58.580857 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1 [sys.catalog]: This master's current role is: LEADER
I20260201 03:08:58.581056 5005 mvcc.cc:204] Tried to move back new op lower bound from 7249573226820255744 to 7249573226534522880. Current Snapshot: MvccSnapshot[applied={T|T < 7249573226820255744}]
I20260201 03:08:58.581305 5004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0867071355a643f59e50d5390fec86b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bcb9810ed2544bbb94c0263ac80809a9" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46111 } } peers { permanent_uuid: "72d8bc3b19e5459caa11176e27435bd1" member_type: VOTER last_known_addr { host: "127.28.181.61" port: 40901 } attrs { promote: false } } peers { permanent_uuid: "0867071355a643f59e50d5390fec86b1" member_type: VOTER last_known_addr { host: "127.28.181.60" port: 38849 } attrs { promote: false } } }
I20260201 03:08:58.581362 5004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8bc3b19e5459caa11176e27435bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:08:58.584170 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:34343 (local address 127.0.0.1:54914)
0201 03:08:58.582347 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.582356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:58.582394 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:58.582675 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:58.582676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:58.582677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:58.582836 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.582837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.583471 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.583471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:58.583967 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.583968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.584044 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.584048 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:08:58.584075 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:58.584105 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":66,"threads_started":1}
I20260201 03:08:58.584434 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46441 (local address 127.0.0.1:53474)
0201 03:08:58.582227 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.582237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:58.582280 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:58.582597 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:58.582598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:58.582600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:58.582802 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.582804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.583508 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.583509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:58.584260 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.584262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.584312 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.584314 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:08:58.584339 (+ 25us) client_negotiation.cc:240] Negotiation successful
0201 03:08:58.584382 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11}
I20260201 03:08:58.584553 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.28.181.2:34343)
0201 03:08:58.582578 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.582579 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.582586 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:58.582608 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.582609 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.582618 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.582674 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:58.582894 (+ 220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.583400 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.584043 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.584412 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.584418 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.584437 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.584460 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":53,"threads_started":1}
I20260201 03:08:58.584841 4718 tablet_service.cc:1505] Processing CreateTablet for tablet 9405de9d3015494a850cdfae81066183 (DEFAULT_TABLE table=test-table [id=08a4cb82a2fd48fbbcb0bdb1d88d7c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:58.584978 4718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9405de9d3015494a850cdfae81066183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:58.585913 5015 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Bootstrap starting.
I20260201 03:08:58.586295 5015 tablet_bootstrap.cc:654] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:58.586355 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.28.181.1:46441)
0201 03:08:58.582484 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.582486 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.582493 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:58.582521 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.582522 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.582529 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.582596 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:58.582875 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.583407 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.585871 (+ 2464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.586265 (+ 394us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.586271 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.586288 (+ 17us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.586312 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":46,"threads_started":1}
I20260201 03:08:58.586553 4582 tablet_service.cc:1505] Processing CreateTablet for tablet 9405de9d3015494a850cdfae81066183 (DEFAULT_TABLE table=test-table [id=08a4cb82a2fd48fbbcb0bdb1d88d7c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:58.586660 4582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9405de9d3015494a850cdfae81066183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:58.586966 5015 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: No bootstrap required, opened a new log
I20260201 03:08:58.587177 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:41021 (local address 127.0.0.1:55318)
0201 03:08:58.582488 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.582498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:58.582547 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:58.582790 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:58.582791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:58.582792 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:58.583034 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.583036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.584555 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.584556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:58.586974 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.586977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.587056 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.587059 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:58.587086 (+ 27us) client_negotiation.cc:240] Negotiation successful
0201 03:08:58.587115 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":47,"threads_started":1}
I20260201 03:08:58.587641 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.582414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.28.181.3:41021)
0201 03:08:58.582529 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.582531 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.582538 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:58.582720 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.582720 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.582727 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.582788 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:58.583499 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.584142 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.587067 (+ 2925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.587539 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.587545 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.587561 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.587584 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260201 03:08:58.587746 5015 ts_tablet_manager.cc:1403] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:08:58.587831 4854 tablet_service.cc:1505] Processing CreateTablet for tablet 9405de9d3015494a850cdfae81066183 (DEFAULT_TABLE table=test-table [id=08a4cb82a2fd48fbbcb0bdb1d88d7c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:08:58.587929 4854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9405de9d3015494a850cdfae81066183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:08:58.587924 5015 raft_consensus.cc:359] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.587977 5015 raft_consensus.cc:385] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:58.587999 5015 raft_consensus.cc:740] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b429af07ea346829b21aec63290f890, State: Initialized, Role: FOLLOWER
I20260201 03:08:58.588058 5015 consensus_queue.cc:260] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.588191 5015 ts_tablet_manager.cc:1434] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:58.588630 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.588336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:49481)
0201 03:08:58.588563 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.588594 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
I20260201 03:08:58.589069 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.588982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:44545)
0201 03:08:58.589020 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.589044 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260201 03:08:58.589218 5020 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Bootstrap starting.
I20260201 03:08:58.589454 5017 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: Bootstrap starting.
I20260201 03:08:58.589478 4785 heartbeater.cc:499] Master 127.28.181.60:38849 was elected leader, sending a full tablet report...
I20260201 03:08:58.589556 5020 tablet_bootstrap.cc:654] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:58.589948 5017 tablet_bootstrap.cc:654] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: Neither blocks nor log segments found. Creating new log.
I20260201 03:08:58.590152 5020 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: No bootstrap required, opened a new log
I20260201 03:08:58.590200 5020 ts_tablet_manager.cc:1403] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:08:58.590322 5020 raft_consensus.cc:359] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.590384 5020 raft_consensus.cc:385] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:58.590402 5020 raft_consensus.cc:740] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1801f0dfc7c44da984786c68e85c06f, State: Initialized, Role: FOLLOWER
I20260201 03:08:58.590444 5020 consensus_queue.cc:260] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.590581 5020 ts_tablet_manager.cc:1434] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260201 03:08:58.590780 5017 tablet_bootstrap.cc:492] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: No bootstrap required, opened a new log
I20260201 03:08:58.590827 5017 ts_tablet_manager.cc:1403] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:08:58.591032 5017 raft_consensus.cc:359] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.591120 5017 raft_consensus.cc:385] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:08:58.591145 5017 raft_consensus.cc:740] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81843fd47754456a914971183b283442, State: Initialized, Role: FOLLOWER
I20260201 03:08:58.591192 5017 consensus_queue.cc:260] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.591300 5017 ts_tablet_manager.cc:1434] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:08:58.591531 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.590767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:53351)
0201 03:08:58.591459 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.591485 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":62,"threads_started":1}
I20260201 03:08:58.591655 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.591422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:43287)
0201 03:08:58.591537 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.591622 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260201 03:08:58.591728 4649 heartbeater.cc:499] Master 127.28.181.60:38849 was elected leader, sending a full tablet report...
I20260201 03:08:58.592074 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.591898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:51003)
0201 03:08:58.592030 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.592051 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260201 03:08:58.592162 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.591851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:44825)
0201 03:08:58.591900 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.592141 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260201 03:08:58.598418 4787 tablet.cc:2378] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:08:58.663311 5023 raft_consensus.cc:493] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:08:58.663395 5023 raft_consensus.cc:515] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.663740 5023 leader_election.cc:290] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81843fd47754456a914971183b283442 (127.28.181.1:46441), 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
I20260201 03:08:58.666286 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.663869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46441 (local address 127.28.181.3:53479)
0201 03:08:58.664029 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.664038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:58.664097 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:58.664196 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:58.664196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:58.664197 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:58.664426 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.664428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.665264 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.665265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:58.666083 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.666084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.666148 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.666153 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:08:58.666187 (+ 34us) client_negotiation.cc:240] Negotiation successful
0201 03:08:58.666221 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260201 03:08:58.666296 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.663862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:34343 (local address 127.28.181.3:41591)
0201 03:08:58.663932 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.663943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:08:58.664012 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:08:58.664183 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:08:58.664184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:08:58.664187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:08:58.664433 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.664434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.665246 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.665246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:08:58.666077 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:08:58.666079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.666166 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.666169 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:08:58.666200 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:08:58.666228 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260201 03:08:58.666677 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.663918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:41591 (local address 127.28.181.2:34343)
0201 03:08:58.663974 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.663976 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.663986 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:08:58.664090 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.664092 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.664102 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.664180 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:58.664507 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.665177 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.666167 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.666563 (+ 396us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.666571 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.666590 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.666618 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:08:58.666733 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.664060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:53479 (local address 127.28.181.1:46441)
0201 03:08:58.664100 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.664102 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.664110 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:08:58.664137 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.664137 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.664146 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.664197 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:08:58.664507 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.665174 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.666160 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.666628 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.666634 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.666650 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.666676 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260201 03:08:58.666860 4602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9405de9d3015494a850cdfae81066183" candidate_uuid: "d1801f0dfc7c44da984786c68e85c06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81843fd47754456a914971183b283442" is_pre_election: true
I20260201 03:08:58.666941 4602 raft_consensus.cc:2468] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1801f0dfc7c44da984786c68e85c06f in term 0.
I20260201 03:08:58.666898 4738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9405de9d3015494a850cdfae81066183" candidate_uuid: "d1801f0dfc7c44da984786c68e85c06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b429af07ea346829b21aec63290f890" is_pre_election: true
I20260201 03:08:58.667012 4738 raft_consensus.cc:2468] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1801f0dfc7c44da984786c68e85c06f in term 0.
I20260201 03:08:58.667081 4808 leader_election.cc:304] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81843fd47754456a914971183b283442, d1801f0dfc7c44da984786c68e85c06f; no voters:
I20260201 03:08:58.667227 5023 raft_consensus.cc:2804] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:08:58.667291 5023 raft_consensus.cc:493] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:08:58.667320 5023 raft_consensus.cc:3060] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:58.667904 5023 raft_consensus.cc:515] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.668044 5023 leader_election.cc:290] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [CANDIDATE]: Term 1 election: Requested vote from peers 81843fd47754456a914971183b283442 (127.28.181.1:46441), 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343)
I20260201 03:08:58.668205 4738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9405de9d3015494a850cdfae81066183" candidate_uuid: "d1801f0dfc7c44da984786c68e85c06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b429af07ea346829b21aec63290f890"
I20260201 03:08:58.668210 4602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9405de9d3015494a850cdfae81066183" candidate_uuid: "d1801f0dfc7c44da984786c68e85c06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81843fd47754456a914971183b283442"
I20260201 03:08:58.668272 4602 raft_consensus.cc:3060] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:58.668273 4738 raft_consensus.cc:3060] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:08:58.668834 4602 raft_consensus.cc:2468] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1801f0dfc7c44da984786c68e85c06f in term 1.
I20260201 03:08:58.668924 4738 raft_consensus.cc:2468] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1801f0dfc7c44da984786c68e85c06f in term 1.
I20260201 03:08:58.669045 4808 leader_election.cc:304] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81843fd47754456a914971183b283442, d1801f0dfc7c44da984786c68e85c06f; no voters:
I20260201 03:08:58.669178 5023 raft_consensus.cc:2804] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:08:58.669317 5023 raft_consensus.cc:697] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 1 LEADER]: Becoming Leader. State: Replica: d1801f0dfc7c44da984786c68e85c06f, State: Running, Role: LEADER
I20260201 03:08:58.669402 4921 heartbeater.cc:499] Master 127.28.181.60:38849 was elected leader, sending a full tablet report...
I20260201 03:08:58.669390 5023 consensus_queue.cc:237] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } }
I20260201 03:08:58.669893 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.669596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:43103)
0201 03:08:58.669624 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.669863 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260201 03:08:58.670045 4453 catalog_manager.cc:5654] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f reported cstate change: term changed from 0 to 1, leader changed from <none> to d1801f0dfc7c44da984786c68e85c06f (127.28.181.3). New cstate: current_term: 1 leader_uuid: "d1801f0dfc7c44da984786c68e85c06f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1801f0dfc7c44da984786c68e85c06f" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20260201 03:08:58.676117 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.674340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.28.181.3:41021)
0201 03:08:58.674379 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0201 03:08:58.674381 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:08:58.674388 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:08:58.674450 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:08:58.674450 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:08:58.674468 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:08:58.674525 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:08:58.674712 (+ 187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.675169 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:08:58.675682 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:08:58.675808 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:08:58.675832 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:08:58.675909 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:08:58.675988 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0201 03:08:58.676039 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:08:58.676057 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260201 03:08:58.676862 4602 raft_consensus.cc:1275] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Refusing update from remote peer d1801f0dfc7c44da984786c68e85c06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:58.676930 4738 raft_consensus.cc:1275] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Refusing update from remote peer d1801f0dfc7c44da984786c68e85c06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:08:58.677059 5023 consensus_queue.cc:1048] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81843fd47754456a914971183b283442" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:58.677145 5028 consensus_queue.cc:1048] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b429af07ea346829b21aec63290f890" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:08:58.677794 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.677670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:56077)
0201 03:08:58.677724 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.677747 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260201 03:08:58.678153 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.678012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:44939)
0201 03:08:58.678106 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.678129 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260201 03:08:58.679066 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.678949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.3:46653)
0201 03:08:58.679021 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.679042 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260201 03:08:58.679464 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.679385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.1:32847)
0201 03:08:58.679411 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.679435 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260201 03:08:58.680282 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.680155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46111 (local address 127.28.181.2:60003)
0201 03:08:58.680238 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.680261 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260201 03:08:58.680853 4453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:58.680939 4453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:08:58.681938 4453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0867071355a643f59e50d5390fec86b1: Sending DeleteTablet for 3 replicas of tablet 9405de9d3015494a850cdfae81066183
I20260201 03:08:58.682129 4582 tablet_service.cc:1552] Processing DeleteTablet for tablet 9405de9d3015494a850cdfae81066183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:58 UTC) from {username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:53474
I20260201 03:08:58.682260 4854 tablet_service.cc:1552] Processing DeleteTablet for tablet 9405de9d3015494a850cdfae81066183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:58 UTC) from {username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:55318
I20260201 03:08:58.682451 4718 tablet_service.cc:1552] Processing DeleteTablet for tablet 9405de9d3015494a850cdfae81066183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:08:58 UTC) from {username='oryx', principal='oryx/127.28.181.60@KRBTEST.COM'} at 127.0.0.1:54914
I20260201 03:08:58.682475 5041 tablet_replica.cc:333] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442: stopping tablet replica
I20260201 03:08:58.682515 5041 raft_consensus.cc:2243] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:58.682461 5042 tablet_replica.cc:333] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: stopping tablet replica
I20260201 03:08:58.682559 5041 raft_consensus.cc:2272] T 9405de9d3015494a850cdfae81066183 P 81843fd47754456a914971183b283442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:58.682619 5042 raft_consensus.cc:2243] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:08:58.682722 5042 raft_consensus.cc:2272] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:58.682724 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 4519
I20260201 03:08:58.683036 5042 ts_tablet_manager.cc:1916] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:58.683431 5043 tablet_replica.cc:333] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: stopping tablet replica
I20260201 03:08:58.683488 5043 raft_consensus.cc:2243] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:08:58.683529 5043 raft_consensus.cc:2272] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:08:58.683828 5043 ts_tablet_manager.cc:1916] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:08:58.684789 5042 ts_tablet_manager.cc:1929] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:58.684881 5042 log.cc:1199] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/9405de9d3015494a850cdfae81066183
I20260201 03:08:58.685189 5042 ts_tablet_manager.cc:1950] T 9405de9d3015494a850cdfae81066183 P d1801f0dfc7c44da984786c68e85c06f: Deleting consensus metadata
I20260201 03:08:58.685307 5043 ts_tablet_manager.cc:1929] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:08:58.685359 5043 log.cc:1199] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/9405de9d3015494a850cdfae81066183
I20260201 03:08:58.685567 4437 catalog_manager.cc:4985] TS d1801f0dfc7c44da984786c68e85c06f (127.28.181.3:41021): tablet 9405de9d3015494a850cdfae81066183 (table test-table [id=08a4cb82a2fd48fbbcb0bdb1d88d7c23]) successfully deleted
I20260201 03:08:58.685662 5043 ts_tablet_manager.cc:1950] T 9405de9d3015494a850cdfae81066183 P 3b429af07ea346829b21aec63290f890: Deleting consensus metadata
I20260201 03:08:58.685933 4439 catalog_manager.cc:4985] TS 3b429af07ea346829b21aec63290f890 (127.28.181.2:34343): tablet 9405de9d3015494a850cdfae81066183 (table test-table [id=08a4cb82a2fd48fbbcb0bdb1d88d7c23]) successfully deleted
I20260201 03:08:58.689260 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:08:58.689169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:46441 (local address 127.0.0.1:53490)
0201 03:08:58.689209 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0201 03:08:58.689229 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:46441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260201 03:08:58.689355 4438 catalog_manager.cc:4712] TS 81843fd47754456a914971183b283442 (127.28.181.1:46441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9405de9d3015494a850cdfae81066183: Network error: Client connection negotiation failed: client connection to 127.28.181.1:46441: connect: Connection refused (error 111)
I20260201 03:08:58.689414 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 4655
I20260201 03:08:58.697227 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 4791
I20260201 03:08:58.701632 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 4132
I20260201 03:08:58.705693 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 4323
2026-02-01T03:08:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8273 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:08:58 dist-test-slave-ck1n krb5kdc[5046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:08:58 dist-test-slave-ck1n krb5kdc[5046](info): set up 2 sockets
Feb 01 03:08:58 dist-test-slave-ck1n krb5kdc[5046](info): commencing operation
krb5kdc: starting...
W20260201 03:09:00.410122 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.684s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:09:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:09:00.515604 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:42935
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:00.580775 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:00.580905 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:00.580924 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:00.582075 5062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:00.582111 5062 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:09:00.582122 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:00.582134 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:00.582144 5062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:00.582154 5062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:00.583415 5062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:42935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:00.583552 5062 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:00.583653 5062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:00.585633 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:00.585693 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:00.585717 5062 server_base.cc:1060] running on GCE node
W20260201 03:09:00.585672 5068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:00.585956 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:00.586150 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:00.587284 5062 hybrid_clock.cc:648] HybridClock initialized: now 1769915340587259 us; error 36 us; skew 500 ppm
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:00.589192 5062 init.cc:377] Logged in from keytab as oryx/127.28.181.62@KRBTEST.COM (short username oryx)
I20260201 03:09:00.589600 5062 webserver.cc:492] Webserver started at http://127.28.181.62:33271/ using document root <none> and password file <none>
I20260201 03:09:00.589800 5062 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:00.589860 5062 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:00.589963 5062 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:00.590792 5062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "aa4c8db1781449c3a8ee7fdb222185e3"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "d9d64d8fec7483511acd7aab1bb015c0"
server_key_iv: "b19a7803b09d20c9c59c845a7452c6c4"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.591096 5062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "aa4c8db1781449c3a8ee7fdb222185e3"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "d9d64d8fec7483511acd7aab1bb015c0"
server_key_iv: "b19a7803b09d20c9c59c845a7452c6c4"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.592074 5062 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:00.592617 5077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:00.592758 5062 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:00.592830 5062 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "aa4c8db1781449c3a8ee7fdb222185e3"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "d9d64d8fec7483511acd7aab1bb015c0"
server_key_iv: "b19a7803b09d20c9c59c845a7452c6c4"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.592895 5062 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:00.601300 5062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:00.603148 5062 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:00.603271 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:00.606748 5062 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:42935
I20260201 03:09:00.606784 5139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:42935 every 8 connection(s)
I20260201 03:09:00.607024 5062 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:00.607498 5140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:00.608393 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5062
I20260201 03:09:00.608482 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:09:00.608626 29396 external_mini_cluster.cc:1468] Setting key f3fc67a5c65ea97b30e75081319a3fea
I20260201 03:09:00.609879 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Bootstrap starting.
I20260201 03:09:00.610342 5140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:00.610555 5140 log.cc:826] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:00.611084 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: No bootstrap required, opened a new log
I20260201 03:09:00.612031 5140 raft_consensus.cc:359] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } }
I20260201 03:09:00.612142 5140 raft_consensus.cc:385] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:00.612180 5140 raft_consensus.cc:740] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c8db1781449c3a8ee7fdb222185e3, State: Initialized, Role: FOLLOWER
I20260201 03:09:00.612282 5140 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } }
I20260201 03:09:00.612344 5140 raft_consensus.cc:399] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:00.612378 5140 raft_consensus.cc:493] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:00.612417 5140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:00.612867 5140 raft_consensus.cc:515] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } }
I20260201 03:09:00.612965 5140 leader_election.cc:304] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa4c8db1781449c3a8ee7fdb222185e3; no voters:
I20260201 03:09:00.613087 5140 leader_election.cc:290] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:09:00.613258 5145 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 1 FOLLOWER]: Leader election won for term 1
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:00.613445 5145 raft_consensus.cc:697] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [term 1 LEADER]: Becoming Leader. State: Replica: aa4c8db1781449c3a8ee7fdb222185e3, State: Running, Role: LEADER
I20260201 03:09:00.613267 5140 sys_catalog.cc:565] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:00.613575 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } }
I20260201 03:09:00.614352 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa4c8db1781449c3a8ee7fdb222185e3. Latest consensus state: current_term: 1 leader_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } } }
I20260201 03:09:00.614432 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:00.614871 5154 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:00.614987 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4c8db1781449c3a8ee7fdb222185e3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 42935 } } }
I20260201 03:09:00.615049 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:00.615346 5154 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:09:00.616535 5154 catalog_manager.cc:1357] Generated new cluster ID: 13003ace8c1a45cbaf3381aaed4fb5f4
I20260201 03:09:00.616616 5154 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:09:00.616890 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.609027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.28.181.62:42935)
0201 03:09:00.609231 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:00.609240 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:00.609256 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:09:00.609607 (+ 351us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:00.609613 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:00.609656 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:00.609787 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:00.610478 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.611105 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.611654 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.611856 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.613835 (+ 1979us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:00.613841 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:00.613842 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:00.613864 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:00.615646 (+ 1782us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.615964 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.615967 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.615967 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.616014 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.616171 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.616171 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.616172 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.616379 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:00.616514 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:00.616625 (+ 111us) server_negotiation.cc:300] Negotiation successful
0201 03:09:00.616726 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":54,"threads_started":1}
I20260201 03:09:00.625398 5154 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:09:00.625842 5154 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:00.629468 5154 catalog_manager.cc:6027] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Generated new TSK 0
I20260201 03:09:00.629598 5154 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:09:00.668252 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:42935
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:00.732618 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:00.732745 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:00.732764 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:00.733978 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:00.734014 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:00.734030 5168 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:09:00.734045 5168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:00.735355 5168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:00.735522 5168 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:00.735679 5168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:00.737645 5176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:00.737659 5173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:00.737656 5174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:00.737898 5168 server_base.cc:1060] running on GCE node
I20260201 03:09:00.738065 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:00.738221 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:00.739341 5168 hybrid_clock.cc:648] HybridClock initialized: now 1769915340739332 us; error 26 us; skew 500 ppm
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:00.741315 5168 init.cc:377] Logged in from keytab as oryx/127.28.181.1@KRBTEST.COM (short username oryx)
I20260201 03:09:00.741755 5168 webserver.cc:492] Webserver started at http://127.28.181.1:39607/ using document root <none> and password file <none>
I20260201 03:09:00.741958 5168 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:00.741989 5168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:00.742085 5168 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:00.742830 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "76d13f1a9394cf93b5f9a3a05ee1a8a0"
server_key_iv: "7d9704dd588211c3f75acfac9d5d1c54"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.743095 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "76d13f1a9394cf93b5f9a3a05ee1a8a0"
server_key_iv: "7d9704dd588211c3f75acfac9d5d1c54"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.744055 5168 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:00.744611 5183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:00.744769 5168 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:00.744822 5168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "76d13f1a9394cf93b5f9a3a05ee1a8a0"
server_key_iv: "7d9704dd588211c3f75acfac9d5d1c54"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.744892 5168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:00.765555 5168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:00.767133 5168 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:00.767220 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:00.767561 5168 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:00.767623 5168 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:00.767666 5168 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:00.767711 5168 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:00.773716 5168 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:42315
I20260201 03:09:00.773741 5298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:42315 every 8 connection(s)
I20260201 03:09:00.774063 5168 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:09:00.775067 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.767920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:38435 (local address 127.28.181.62:42935)
0201 03:09:00.767976 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:00.767978 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:00.768121 (+ 143us) server_negotiation.cc:408] Connection header received
0201 03:09:00.768480 (+ 359us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:00.768482 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:00.768494 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:00.768564 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:00.769361 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.770113 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.770797 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.771247 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.772642 (+ 1395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:00.772645 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:00.772645 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:00.772654 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:00.774135 (+ 1481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.774529 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.774530 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.774530 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.774549 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.774680 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.774680 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.774680 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.774743 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:00.774780 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:00.774954 (+ 174us) server_negotiation.cc:300] Negotiation successful
0201 03:09:00.775011 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:09:00.775158 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.767892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.1:38435)
0201 03:09:00.768012 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:00.768036 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:00.768401 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:00.768604 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:00.768607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:00.768888 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:00.769262 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.769266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.770192 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.770192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:00.770712 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.770715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.770837 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.771244 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:00.771264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:00.772560 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:00.774203 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.774206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.774208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.774454 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.774604 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.774605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.774605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.774649 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.774781 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:00.774784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:00.774879 (+ 95us) client_negotiation.cc:769] Sending connection context
0201 03:09:00.774946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:09:00.775023 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":70,"threads_started":1}
I20260201 03:09:00.775580 5299 heartbeater.cc:344] Connected to a master server at 127.28.181.62:42935
I20260201 03:09:00.775665 5299 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:00.775820 5299 heartbeater.cc:507] Master 127.28.181.62:42935 requested a full tablet report, sending...
I20260201 03:09:00.776226 5093 ts_manager.cc:194] Registered new tserver with Master: b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315)
I20260201 03:09:00.776788 5093 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:38435
I20260201 03:09:00.780030 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.776350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:50137 (local address 127.28.181.62:42935)
0201 03:09:00.776381 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:00.776384 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:00.776493 (+ 109us) server_negotiation.cc:408] Connection header received
0201 03:09:00.776554 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:00.776554 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:00.776562 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:00.776626 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:00.776990 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.777381 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.777814 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.777901 (+ 87us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.778357 (+ 456us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:00.778359 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:00.778359 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:00.778364 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:00.779450 (+ 1086us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.779649 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.779650 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.779650 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.779665 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.779763 (+ 98us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.779763 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.779764 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.779817 (+ 53us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:00.779852 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:00.779940 (+ 88us) server_negotiation.cc:300] Negotiation successful
0201 03:09:00.779986 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:09:00.780035 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.776307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.1:50137)
0201 03:09:00.776432 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:00.776440 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:00.776505 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:00.776638 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:00.776639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:00.776752 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:00.776940 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.776941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.777448 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.777448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:00.777721 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.777722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.777808 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.778117 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:00.778120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:00.778297 (+ 177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:00.779506 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.779506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.779506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.779602 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.779707 (+ 105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.779707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.779707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.779733 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.779858 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:00.779858 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:00.779883 (+ 25us) client_negotiation.cc:769] Sending connection context
0201 03:09:00.779924 (+ 41us) client_negotiation.cc:240] Negotiation successful
0201 03:09:00.779978 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260201 03:09:00.781220 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5168
I20260201 03:09:00.781309 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:09:00.781450 29396 external_mini_cluster.cc:1468] Setting key 5cfb1530b9bee5b99fd3898a74cb828a
WARNING: no policy specified for oryx/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:09:00.815912 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:42935
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:00.880046 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:00.880167 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:00.880185 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:00.881345 5310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:00.881382 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:00.881399 5310 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:09:00.881446 5310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:00.882783 5310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:00.882921 5310 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:00.883080 5310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:09:00.885146 5310 server_base.cc:1060] running on GCE node
W20260201 03:09:00.885136 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:00.885150 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:00.885136 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:00.885406 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:00.885582 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:00.886715 5310 hybrid_clock.cc:648] HybridClock initialized: now 1769915340886704 us; error 21 us; skew 500 ppm
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:00.888669 5310 init.cc:377] Logged in from keytab as oryx/127.28.181.2@KRBTEST.COM (short username oryx)
I20260201 03:09:00.889098 5310 webserver.cc:492] Webserver started at http://127.28.181.2:37985/ using document root <none> and password file <none>
I20260201 03:09:00.889262 5310 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:00.889309 5310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:00.889413 5310 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:00.890198 5310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "9800170294934b47b54d0cb122820ec0"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "2be0fc4e974af5ec3b34d3208529ab5e"
server_key_iv: "33483f1da2562889e62b02cc6f9cd013"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.890463 5310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "9800170294934b47b54d0cb122820ec0"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "2be0fc4e974af5ec3b34d3208529ab5e"
server_key_iv: "33483f1da2562889e62b02cc6f9cd013"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.891427 5310 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:00.891968 5325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:00.892107 5310 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:00.892180 5310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "9800170294934b47b54d0cb122820ec0"
format_stamp: "Formatted at 2026-02-01 03:09:00 on dist-test-slave-ck1n"
server_key: "2be0fc4e974af5ec3b34d3208529ab5e"
server_key_iv: "33483f1da2562889e62b02cc6f9cd013"
server_key_version: "encryptionkey@0"
I20260201 03:09:00.892247 5310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:00.901880 5310 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:00.903458 5310 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:00.903581 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:00.903980 5310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:00.904033 5310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:00.904059 5310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:00.904114 5310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:09:00 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:00.910209 5310 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:36313
I20260201 03:09:00.910251 5440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:36313 every 8 connection(s)
I20260201 03:09:00.910534 5310 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:09:00.911095 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.904314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:48335 (local address 127.28.181.62:42935)
0201 03:09:00.904354 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:00.904357 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:00.904493 (+ 136us) server_negotiation.cc:408] Connection header received
0201 03:09:00.904883 (+ 390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:00.904884 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:00.904895 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:00.904937 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:00.905669 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.906173 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.906925 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.907080 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.908625 (+ 1545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:00.908629 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:00.908629 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:00.908636 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:00.910022 (+ 1386us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.910411 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.910412 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.910412 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.910437 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.910591 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.910592 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.910592 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.910690 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:00.910760 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:00.910977 (+ 217us) server_negotiation.cc:300] Negotiation successful
0201 03:09:00.911039 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260201 03:09:00.911141 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.904295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.2:48335)
0201 03:09:00.904404 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:00.904434 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:00.904790 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:00.904978 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:00.904985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:00.905247 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:00.905595 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.905599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.906272 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.906273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:00.906847 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.906850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.906984 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.907410 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:00.907427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:00.908536 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:00.910096 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.910099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.910100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.910340 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.910495 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.910495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.910495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.910537 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.910755 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:00.910758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:00.910873 (+ 115us) client_negotiation.cc:769] Sending connection context
0201 03:09:00.910954 (+ 81us) client_negotiation.cc:240] Negotiation successful
0201 03:09:00.911031 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":66,"threads_started":1}
I20260201 03:09:00.911633 5441 heartbeater.cc:344] Connected to a master server at 127.28.181.62:42935
I20260201 03:09:00.911700 5441 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:00.911875 5441 heartbeater.cc:507] Master 127.28.181.62:42935 requested a full tablet report, sending...
I20260201 03:09:00.912168 5094 ts_manager.cc:194] Registered new tserver with Master: 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313)
I20260201 03:09:00.912627 5094 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:48335
I20260201 03:09:00.916436 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.912473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:48925 (local address 127.28.181.62:42935)
0201 03:09:00.912513 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:00.912515 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:00.912615 (+ 100us) server_negotiation.cc:408] Connection header received
0201 03:09:00.912671 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:00.912672 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:00.912701 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:00.912749 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:00.913107 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.913483 (+ 376us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.913986 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.914145 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.914625 (+ 480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:00.914628 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:00.914628 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:00.914633 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:00.915611 (+ 978us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.915888 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.915889 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.915889 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.915908 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:00.916047 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:00.916048 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:00.916049 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:00.916132 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:00.916193 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:00.916307 (+ 114us) server_negotiation.cc:300] Negotiation successful
0201 03:09:00.916365 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260201 03:09:00.916450 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:00.912415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.2:48925)
0201 03:09:00.912563 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:00.912571 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:00.912624 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:00.912787 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:00.912788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:00.912893 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:00.913028 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.913029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.913558 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:00.913559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:00.913918 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:00.913920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:00.913989 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:00.914309 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:00.914313 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:00.914553 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:00.915693 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.915693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.915694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.915815 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.915974 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:00.915975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:00.915976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:00.916001 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:00.916188 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:00.916189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:00.916223 (+ 34us) client_negotiation.cc:769] Sending connection context
0201 03:09:00.916281 (+ 58us) client_negotiation.cc:240] Negotiation successful
0201 03:09:00.916363 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260201 03:09:00.918759 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5310
I20260201 03:09:00.918851 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:09:00.919008 29396 external_mini_cluster.cc:1468] Setting key 01cad664bd60dfc6111ef90aaf038174
WARNING: no policy specified for oryx/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab.
I20260201 03:09:00.953541 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:42935
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:01.018261 5452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:01.018422 5452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:01.018455 5452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:01.019588 5452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:01.019645 5452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:01.019676 5452 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:09:01.019726 5452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:01.021013 5452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.181.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:01.021199 5452 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:01.021351 5452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:01.023451 5460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:01.023435 5457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:01.023505 5458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:01.023505 5452 server_base.cc:1060] running on GCE node
I20260201 03:09:01.023720 5452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:01.023901 5452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:01.025024 5452 hybrid_clock.cc:648] HybridClock initialized: now 1769915341025016 us; error 25 us; skew 500 ppm
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:01.026968 5452 init.cc:377] Logged in from keytab as oryx/127.28.181.3@KRBTEST.COM (short username oryx)
I20260201 03:09:01.027377 5452 webserver.cc:492] Webserver started at http://127.28.181.3:43117/ using document root <none> and password file <none>
I20260201 03:09:01.027530 5452 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:01.027576 5452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:01.027680 5452 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:01.028429 5452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "45293ded2bb2499b8c811f19f7d0d0be"
format_stamp: "Formatted at 2026-02-01 03:09:01 on dist-test-slave-ck1n"
server_key: "fc7c613eb4b25940d2237a967ce6b8c4"
server_key_iv: "a1f41fa0bd18d639e4bfcd2f83c2a309"
server_key_version: "encryptionkey@0"
I20260201 03:09:01.028699 5452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "45293ded2bb2499b8c811f19f7d0d0be"
format_stamp: "Formatted at 2026-02-01 03:09:01 on dist-test-slave-ck1n"
server_key: "fc7c613eb4b25940d2237a967ce6b8c4"
server_key_iv: "a1f41fa0bd18d639e4bfcd2f83c2a309"
server_key_version: "encryptionkey@0"
I20260201 03:09:01.029670 5452 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:01.030269 5467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.030413 5452 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.030488 5452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "45293ded2bb2499b8c811f19f7d0d0be"
format_stamp: "Formatted at 2026-02-01 03:09:01 on dist-test-slave-ck1n"
server_key: "fc7c613eb4b25940d2237a967ce6b8c4"
server_key_iv: "a1f41fa0bd18d639e4bfcd2f83c2a309"
server_key_version: "encryptionkey@0"
I20260201 03:09:01.030550 5452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:01.035705 5452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:01.037236 5452 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:01.037354 5452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:01.037740 5452 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:01.037808 5452 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.037845 5452 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:01.037875 5452 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:01.043761 5452 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:42891
I20260201 03:09:01.043845 5582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:42891 every 8 connection(s)
I20260201 03:09:01.044080 5452 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:09:01.045215 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.038096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:34955 (local address 127.28.181.62:42935)
0201 03:09:01.038153 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.038156 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.038277 (+ 121us) server_negotiation.cc:408] Connection header received
0201 03:09:01.038591 (+ 314us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.038592 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.038602 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.038671 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.039253 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.039952 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.040658 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.041126 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.042821 (+ 1695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:01.042824 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:01.042824 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:01.042837 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:01.044317 (+ 1480us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.044670 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.044671 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.044672 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.044693 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.044831 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.044831 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.044831 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.044897 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:01.044935 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.045101 (+ 166us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.045161 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260201 03:09:01.045307 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.038065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.3:34955)
0201 03:09:01.038183 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.038206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.038522 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.038702 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.038705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.038908 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:01.039169 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.039172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.040042 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.040043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.040580 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.040582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.040700 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.041130 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:01.041148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:01.042730 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:01.044405 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.044408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.044410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.044605 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.044749 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.044749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.044749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.044791 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.044938 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:01.044941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:01.045019 (+ 78us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.045089 (+ 70us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.045164 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":67,"threads_started":1}
I20260201 03:09:01.045691 5583 heartbeater.cc:344] Connected to a master server at 127.28.181.62:42935
I20260201 03:09:01.045761 5583 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:01.045953 5583 heartbeater.cc:507] Master 127.28.181.62:42935 requested a full tablet report, sending...
I20260201 03:09:01.046289 5093 ts_manager.cc:194] Registered new tserver with Master: 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.046411 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5452
I20260201 03:09:01.046499 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:09:01.046659 29396 external_mini_cluster.cc:1468] Setting key d6564b149e98736af80950bc56cc92ee
I20260201 03:09:01.046730 5093 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:34955
I20260201 03:09:01.047466 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:09:01.051396 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.046655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.28.181.3:56499)
0201 03:09:01.046856 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.046866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.046935 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.047143 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.047143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.047282 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:01.047454 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.047456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.048066 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.048067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.048409 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.048410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.048481 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.048895 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:01.048898 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:01.049111 (+ 213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:01.050662 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.050663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.050663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.050776 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.050955 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.050956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.050957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.050994 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.051204 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:01.051205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:01.051232 (+ 27us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.051277 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.051337 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260201 03:09:01.051457 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.046595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:56499 (local address 127.28.181.62:42935)
0201 03:09:01.046629 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.046630 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.046913 (+ 283us) server_negotiation.cc:408] Connection header received
0201 03:09:01.047010 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.047011 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.047020 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.047068 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.047524 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.047997 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.048479 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.048658 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.049235 (+ 577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:01.049238 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:01.049238 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:01.049245 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:01.050589 (+ 1344us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.050841 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.050842 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.050843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.050867 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.051047 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.051048 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.051048 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.051147 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:01.051210 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.051298 (+ 88us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.051368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:09:01.065344 5143 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:01.061272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.28.181.62:42935)
0201 03:09:01.061320 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.061322 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.061330 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:01.061384 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.061384 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.061392 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.061453 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.062056 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.062516 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.062996 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.063147 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.065280 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54456: BlockingRecv error: recv got EOF from 127.0.0.1:54456 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:01.070686 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.066046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.28.181.62:42935)
0201 03:09:01.066082 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.066083 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.066136 (+ 53us) server_negotiation.cc:408] Connection header received
0201 03:09:01.066179 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.066180 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.066186 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.066229 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.066672 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.067054 (+ 382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.067437 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.067518 (+ 81us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.069067 (+ 1549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:01.069080 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:01.069080 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:01.069084 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:01.070093 (+ 1009us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.070318 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.070318 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.070318 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.070333 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.070426 (+ 93us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.070426 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.070426 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.070481 (+ 55us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:01.070535 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.070598 (+ 63us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.070642 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260201 03:09:01.071794 5093 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:09:01.072955 5093 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:01.077324 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36313 (local address 127.0.0.1:48540)
0201 03:09:01.074931 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.074941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.074997 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.075263 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.075264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.075265 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.075388 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.075389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076373 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.076374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.077106 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.077108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.077162 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.077184 (+ 22us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.077224 (+ 40us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.077261 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":70,"threads_started":1}
I20260201 03:09:01.077375 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:42891 (local address 127.0.0.1:38296)
0201 03:09:01.074975 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.074983 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.075032 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.075189 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.075190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.075191 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.075386 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.075388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076494 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.076495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.077187 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.077189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.077243 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.077245 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.077271 (+ 26us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.077296 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":75,"threads_started":1}
I20260201 03:09:01.077795 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:42315 (local address 127.0.0.1:51928)
0201 03:09:01.074894 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.074922 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.074994 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.075515 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.075516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.075517 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.075671 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.075672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076805 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.076806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.077334 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.077336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.077644 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.077647 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.077686 (+ 39us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.077717 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":45,"threads_started":1}
I20260201 03:09:01.077883 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.28.181.3:42891)
0201 03:09:01.074992 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.074999 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.075025 (+ 26us) server_negotiation.cc:408] Connection header received
0201 03:09:01.075090 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.075092 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.075104 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.075155 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.075464 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076432 (+ 968us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.077274 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.077698 (+ 424us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.077737 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.077760 (+ 23us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.077814 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260201 03:09:01.078135 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.28.181.2:36313)
0201 03:09:01.075108 (+ 333us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.075113 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.075120 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:01.075166 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.075168 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.075181 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.075243 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.075458 (+ 215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076279 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.077445 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.077981 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.078015 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.078039 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.078072 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":88,"threads_started":1}
I20260201 03:09:01.078328 5517 tablet_service.cc:1505] Processing CreateTablet for tablet 7250801365ba4a5cb7b48411766db022 (DEFAULT_TABLE table=test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.078580 5517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7250801365ba4a5cb7b48411766db022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.078632 5375 tablet_service.cc:1505] Processing CreateTablet for tablet 7250801365ba4a5cb7b48411766db022 (DEFAULT_TABLE table=test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.078714 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.074846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.28.181.1:42315)
0201 03:09:01.075370 (+ 524us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.075377 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.075385 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:01.075418 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.075420 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.075435 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.075725 (+ 290us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.075807 (+ 82us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.076717 (+ 910us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.078145 (+ 1428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.078560 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.078593 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.078617 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.078652 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":69,"threads_started":1}
I20260201 03:09:01.078878 5375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7250801365ba4a5cb7b48411766db022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.079101 5233 tablet_service.cc:1505] Processing CreateTablet for tablet 7250801365ba4a5cb7b48411766db022 (DEFAULT_TABLE table=test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.079334 5233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7250801365ba4a5cb7b48411766db022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.080535 5608 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Bootstrap starting.
I20260201 03:09:01.081050 5608 tablet_bootstrap.cc:654] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.081275 5609 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Bootstrap starting.
I20260201 03:09:01.081333 5608 log.cc:826] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:01.081681 5611 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Bootstrap starting.
I20260201 03:09:01.081881 5609 tablet_bootstrap.cc:654] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.082181 5609 log.cc:826] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:01.082253 5611 tablet_bootstrap.cc:654] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.082275 5608 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: No bootstrap required, opened a new log
I20260201 03:09:01.082324 5608 ts_tablet_manager.cc:1403] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:01.082726 5611 log.cc:826] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:01.083003 5609 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: No bootstrap required, opened a new log
I20260201 03:09:01.083069 5609 ts_tablet_manager.cc:1403] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260201 03:09:01.083370 5611 tablet_bootstrap.cc:492] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: No bootstrap required, opened a new log
I20260201 03:09:01.083461 5611 ts_tablet_manager.cc:1403] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:01.083752 5608 raft_consensus.cc:359] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.083869 5608 raft_consensus.cc:385] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.083917 5608 raft_consensus.cc:740] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45293ded2bb2499b8c811f19f7d0d0be, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.084023 5608 consensus_queue.cc:260] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.084336 5583 heartbeater.cc:499] Master 127.28.181.62:42935 was elected leader, sending a full tablet report...
I20260201 03:09:01.084422 5609 raft_consensus.cc:359] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.084327 5608 ts_tablet_manager.cc:1434] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:09:01.084590 5609 raft_consensus.cc:385] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.084657 5609 raft_consensus.cc:740] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.084757 5609 consensus_queue.cc:260] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.084793 5611 raft_consensus.cc:359] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.084932 5611 raft_consensus.cc:385] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.085000 5611 raft_consensus.cc:740] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.085026 5609 ts_tablet_manager.cc:1434] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:09:01.085129 5611 consensus_queue.cc:260] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.085234 5441 heartbeater.cc:499] Master 127.28.181.62:42935 was elected leader, sending a full tablet report...
I20260201 03:09:01.085335 5611 ts_tablet_manager.cc:1434] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:09:01.085376 5299 heartbeater.cc:499] Master 127.28.181.62:42935 was elected leader, sending a full tablet report...
W20260201 03:09:01.161095 5442 tablet.cc:2378] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:09:01.274639 5300 tablet.cc:2378] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:09:01.281571 5615 raft_consensus.cc:493] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:01.281646 5615 raft_consensus.cc:515] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.281911 5615 leader_election.cc:290] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.283743 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.282044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:42315 (local address 127.28.181.2:60367)
0201 03:09:01.282092 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.282102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.282156 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.282312 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.282313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.282315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.282509 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.282510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283070 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.283071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.283579 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.283580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283628 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.283631 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.283646 (+ 15us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.283667 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260201 03:09:01.283834 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.282064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:42891 (local address 127.28.181.2:44505)
0201 03:09:01.282218 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.282227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.282287 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.282406 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.282406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.282408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.282589 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.282590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283149 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.283150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.283653 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.283654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283709 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.283712 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.283740 (+ 28us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.283768 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260201 03:09:01.284132 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.282077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:60367 (local address 127.28.181.1:42315)
0201 03:09:01.282115 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.282117 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.282141 (+ 24us) server_negotiation.cc:408] Connection header received
0201 03:09:01.282230 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.282231 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.282243 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.282294 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.282569 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283004 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.283669 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.284053 (+ 384us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.284060 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.284076 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.284095 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:09:01.284248 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.282145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:44505 (local address 127.28.181.3:42891)
0201 03:09:01.282185 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.282187 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.282298 (+ 111us) server_negotiation.cc:408] Connection header received
0201 03:09:01.282334 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.282335 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.282343 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.282398 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.282654 (+ 256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.283082 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.283725 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.284122 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.284128 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.284144 (+ 16us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.284181 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:09:01.284397 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7250801365ba4a5cb7b48411766db022" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" is_pre_election: true
I20260201 03:09:01.284438 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7250801365ba4a5cb7b48411766db022" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be" is_pre_election: true
I20260201 03:09:01.284520 5537 raft_consensus.cc:2468] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.284521 5253 raft_consensus.cc:2468] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.284687 5328 leader_election.cc:304] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9800170294934b47b54d0cb122820ec0, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.284806 5615 raft_consensus.cc:2804] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:01.284859 5615 raft_consensus.cc:493] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:01.284875 5615 raft_consensus.cc:3060] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.285449 5615 raft_consensus.cc:515] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.285578 5615 leader_election.cc:290] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Requested vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.285722 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7250801365ba4a5cb7b48411766db022" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be"
I20260201 03:09:01.285720 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7250801365ba4a5cb7b48411766db022" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
I20260201 03:09:01.285801 5253 raft_consensus.cc:3060] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.285806 5537 raft_consensus.cc:3060] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.286500 5253 raft_consensus.cc:2468] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.286501 5537 raft_consensus.cc:2468] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.286664 5327 leader_election.cc:304] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, 9800170294934b47b54d0cb122820ec0; no voters:
I20260201 03:09:01.286758 5615 raft_consensus.cc:2804] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:01.286886 5615 raft_consensus.cc:697] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Becoming Leader. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Running, Role: LEADER
I20260201 03:09:01.286967 5615 consensus_queue.cc:237] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.287586 5093 catalog_manager.cc:5654] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9800170294934b47b54d0cb122820ec0 (127.28.181.2). New cstate: current_term: 1 leader_uuid: "9800170294934b47b54d0cb122820ec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } health_report { overall_health: UNKNOWN } } }
W20260201 03:09:01.294571 5584 tablet.cc:2378] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:09:01.301612 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.299952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.28.181.2:36313)
0201 03:09:01.299989 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.299990 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.299998 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:01.300057 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.300058 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.300071 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.300125 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.300331 (+ 206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.300712 (+ 381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.301206 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.301329 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.301351 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.301418 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.301484 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.301531 (+ 47us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.301550 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260201 03:09:01.302881 5537 raft_consensus.cc:1275] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.302903 5253 raft_consensus.cc:1275] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.303189 5615 consensus_queue.cc:1048] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.303309 5618 consensus_queue.cc:1048] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.305086 5621 mvcc.cc:204] Tried to move back new op lower bound from 7249573237974560768 to 7249573237911801856. Current Snapshot: MvccSnapshot[applied={T|T < 7249573237974560768}]
I20260201 03:09:01.305212 5622 mvcc.cc:204] Tried to move back new op lower bound from 7249573237974560768 to 7249573237911801856. Current Snapshot: MvccSnapshot[applied={T|T < 7249573237974560768}]
I20260201 03:09:01.305881 5623 mvcc.cc:204] Tried to move back new op lower bound from 7249573237974560768 to 7249573237911801856. Current Snapshot: MvccSnapshot[applied={T|T < 7249573237974560768}]
I20260201 03:09:01.306840 5094 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.306938 5094 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.307482 5094 catalog_manager.cc:5941] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Sending DeleteTablet for 3 replicas of tablet 7250801365ba4a5cb7b48411766db022
I20260201 03:09:01.307736 5233 tablet_service.cc:1552] Processing DeleteTablet for tablet 7250801365ba4a5cb7b48411766db022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:51928
I20260201 03:09:01.307746 5375 tablet_service.cc:1552] Processing DeleteTablet for tablet 7250801365ba4a5cb7b48411766db022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:48540
I20260201 03:09:01.307847 5517 tablet_service.cc:1552] Processing DeleteTablet for tablet 7250801365ba4a5cb7b48411766db022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260201 03:09:01.307915 5632 tablet_replica.cc:333] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: stopping tablet replica
I20260201 03:09:01.308010 5632 raft_consensus.cc:2243] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:09:01.308033 5633 tablet_replica.cc:333] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: stopping tablet replica
I20260201 03:09:01.308148 5633 raft_consensus.cc:2243] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.308144 5632 raft_consensus.cc:2272] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.308156 5094 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260201 03:09:01.308261 5633 raft_consensus.cc:2272] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus is shut down!
W20260201 03:09:01.308269 5094 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:01.308514 5632 ts_tablet_manager.cc:1916] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.308588 5633 ts_tablet_manager.cc:1916] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.309484 5233 tablet_service.cc:1505] Processing CreateTablet for tablet 83c9f34afa254ee389581319af2b0c4e (DEFAULT_TABLE table=test-table [id=b1c68586d13940d7b8a3170cf9bfb265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.309571 5233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c9f34afa254ee389581319af2b0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.310003 5632 ts_tablet_manager.cc:1929] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.310025 5633 ts_tablet_manager.cc:1929] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.310050 5632 log.cc:1199] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/7250801365ba4a5cb7b48411766db022
I20260201 03:09:01.310074 5633 log.cc:1199] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/7250801365ba4a5cb7b48411766db022
I20260201 03:09:01.310287 5632 ts_tablet_manager.cc:1950] T 7250801365ba4a5cb7b48411766db022 P 9800170294934b47b54d0cb122820ec0: Deleting consensus metadata
I20260201 03:09:01.310308 5633 ts_tablet_manager.cc:1950] T 7250801365ba4a5cb7b48411766db022 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting consensus metadata
I20260201 03:09:01.310757 5611 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Bootstrap starting.
I20260201 03:09:01.310920 5078 catalog_manager.cc:4985] TS 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313): tablet 7250801365ba4a5cb7b48411766db022 (table test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]) successfully deleted
I20260201 03:09:01.311102 5375 tablet_service.cc:1505] Processing CreateTablet for tablet 83c9f34afa254ee389581319af2b0c4e (DEFAULT_TABLE table=test-table [id=b1c68586d13940d7b8a3170cf9bfb265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.311136 5611 tablet_bootstrap.cc:654] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.311179 5375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c9f34afa254ee389581319af2b0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.311724 5611 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: No bootstrap required, opened a new log
I20260201 03:09:01.311759 5611 ts_tablet_manager.cc:1403] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:01.311892 5611 raft_consensus.cc:359] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.311960 5611 raft_consensus.cc:385] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.311977 5611 raft_consensus.cc:740] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.312016 5611 consensus_queue.cc:260] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.312085 5611 ts_tablet_manager.cc:1434] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.312278 5609 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Bootstrap starting.
I20260201 03:09:01.312547 5079 catalog_manager.cc:4985] TS 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891): tablet 7250801365ba4a5cb7b48411766db022 (table test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]) successfully deleted
I20260201 03:09:01.312644 5609 tablet_bootstrap.cc:654] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.312677 5517 tablet_service.cc:1505] Processing CreateTablet for tablet 83c9f34afa254ee389581319af2b0c4e (DEFAULT_TABLE table=test-table [id=b1c68586d13940d7b8a3170cf9bfb265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.312757 5517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c9f34afa254ee389581319af2b0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.313226 5609 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: No bootstrap required, opened a new log
I20260201 03:09:01.313261 5609 ts_tablet_manager.cc:1403] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:09:01.313371 5609 raft_consensus.cc:359] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.313414 5609 raft_consensus.cc:385] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.313429 5609 raft_consensus.cc:740] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.313468 5609 consensus_queue.cc:260] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.313542 5609 ts_tablet_manager.cc:1434] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.313740 5608 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Bootstrap starting.
I20260201 03:09:01.314127 5608 tablet_bootstrap.cc:654] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.314682 5631 tablet_replica.cc:333] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: stopping tablet replica
I20260201 03:09:01.314750 5631 raft_consensus.cc:2243] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.314837 5631 raft_consensus.cc:2272] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.314952 5608 tablet_bootstrap.cc:492] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: No bootstrap required, opened a new log
I20260201 03:09:01.314993 5608 ts_tablet_manager.cc:1403] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:01.315116 5608 raft_consensus.cc:359] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.315148 5608 raft_consensus.cc:385] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.315160 5608 raft_consensus.cc:740] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45293ded2bb2499b8c811f19f7d0d0be, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.315209 5608 consensus_queue.cc:260] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.315222 5631 ts_tablet_manager.cc:1916] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.315276 5608 ts_tablet_manager.cc:1434] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.316452 5631 ts_tablet_manager.cc:1929] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.316514 5631 log.cc:1199] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/7250801365ba4a5cb7b48411766db022
I20260201 03:09:01.316705 5631 ts_tablet_manager.cc:1950] T 7250801365ba4a5cb7b48411766db022 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting consensus metadata
I20260201 03:09:01.317014 5080 catalog_manager.cc:4985] TS b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315): tablet 7250801365ba4a5cb7b48411766db022 (table test-table [id=4c8cc7bcd20b4a2280c9edd87cf953a9]) successfully deleted
I20260201 03:09:01.394096 5627 raft_consensus.cc:493] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:01.394188 5627 raft_consensus.cc:515] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.394351 5627 leader_election.cc:290] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.394490 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c9f34afa254ee389581319af2b0c4e" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be" is_pre_election: true
I20260201 03:09:01.394498 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c9f34afa254ee389581319af2b0c4e" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" is_pre_election: true
I20260201 03:09:01.394584 5253 raft_consensus.cc:2468] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.394585 5537 raft_consensus.cc:2468] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.394699 5327 leader_election.cc:304] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, 9800170294934b47b54d0cb122820ec0; no voters:
I20260201 03:09:01.394778 5627 raft_consensus.cc:2804] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:01.394830 5627 raft_consensus.cc:493] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:01.394858 5627 raft_consensus.cc:3060] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.395637 5627 raft_consensus.cc:515] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.395754 5627 leader_election.cc:290] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Requested vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.395892 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c9f34afa254ee389581319af2b0c4e" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be"
I20260201 03:09:01.395913 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c9f34afa254ee389581319af2b0c4e" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
I20260201 03:09:01.395964 5537 raft_consensus.cc:3060] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.395982 5253 raft_consensus.cc:3060] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.396530 5253 raft_consensus.cc:2468] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.396623 5537 raft_consensus.cc:2468] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.396682 5328 leader_election.cc:304] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9800170294934b47b54d0cb122820ec0, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.396752 5627 raft_consensus.cc:2804] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:01.396811 5627 raft_consensus.cc:697] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Becoming Leader. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Running, Role: LEADER
I20260201 03:09:01.396867 5627 consensus_queue.cc:237] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.397361 5094 catalog_manager.cc:5654] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9800170294934b47b54d0cb122820ec0 (127.28.181.2). New cstate: current_term: 1 leader_uuid: "9800170294934b47b54d0cb122820ec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } health_report { overall_health: UNKNOWN } } }
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915340, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.62@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:01.427698 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.422396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:42935 (local address 127.0.0.1:54476)
0201 03:09:01.422538 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.422554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.422629 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.422751 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.422752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.423032 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:01.423166 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.423168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.423709 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.423709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.424022 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.424023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.424101 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.424608 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:01.424612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:01.425720 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:01.426893 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.426898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.426900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.427046 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.427200 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:01.427201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:01.427202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:01.427227 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:01.427395 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:01.427399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:01.427435 (+ 36us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.427494 (+ 59us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.427584 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260201 03:09:01.427699 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.422463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.28.181.62:42935)
0201 03:09:01.422524 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.422527 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.422614 (+ 87us) server_negotiation.cc:408] Connection header received
0201 03:09:01.422690 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.422691 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.422699 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.422746 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.423246 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.423641 (+ 395us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.424100 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.424249 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.425812 (+ 1563us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:01.425816 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:01.425816 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:01.425822 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:01.426789 (+ 967us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.427123 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.427124 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.427124 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.427147 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.427273 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.427273 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.427274 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.427343 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:01.427403 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.427515 (+ 112us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.427623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":40}
I20260201 03:09:01.429162 5094 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:54476:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:09:01.429318 5094 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:01.430429 5375 tablet_service.cc:1505] Processing CreateTablet for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7faf93c8dd104334952f5b3bfab07426]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:09:01.430485 5517 tablet_service.cc:1505] Processing CreateTablet for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7faf93c8dd104334952f5b3bfab07426]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:09:01.430496 5233 tablet_service.cc:1505] Processing CreateTablet for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7faf93c8dd104334952f5b3bfab07426]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:09:01.430557 5375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.430598 5517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.430598 5233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21db0dc4ae2a47f9ab39b1beacdd9bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.431574 5609 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: Bootstrap starting.
I20260201 03:09:01.431738 5608 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: Bootstrap starting.
I20260201 03:09:01.431738 5611 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: Bootstrap starting.
I20260201 03:09:01.431890 5609 tablet_bootstrap.cc:654] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.432174 5608 tablet_bootstrap.cc:654] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.432266 5611 tablet_bootstrap.cc:654] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.432368 5609 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: No bootstrap required, opened a new log
I20260201 03:09:01.432426 5609 ts_tablet_manager.cc:1403] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:09:01.432531 5609 raft_consensus.cc:359] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.432585 5609 raft_consensus.cc:385] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.432606 5609 raft_consensus.cc:740] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.432657 5609 consensus_queue.cc:260] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.432731 5609 ts_tablet_manager.cc:1434] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.432801 5608 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: No bootstrap required, opened a new log
I20260201 03:09:01.432842 5608 ts_tablet_manager.cc:1403] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:01.432739 5627 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.432893 5627 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.432993 5608 raft_consensus.cc:359] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.433043 5608 raft_consensus.cc:385] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.433059 5608 raft_consensus.cc:740] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45293ded2bb2499b8c811f19f7d0d0be, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.433095 5608 consensus_queue.cc:260] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.433168 5608 ts_tablet_manager.cc:1434] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.433247 5614 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.433312 5614 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.433554 5611 tablet_bootstrap.cc:492] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: No bootstrap required, opened a new log
I20260201 03:09:01.433601 5611 ts_tablet_manager.cc:1403] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:01.433761 5611 raft_consensus.cc:359] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.433851 5611 raft_consensus.cc:385] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.433871 5611 raft_consensus.cc:740] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.433916 5611 consensus_queue.cc:260] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.433995 5611 ts_tablet_manager.cc:1434] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.434037 5616 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.434098 5616 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.501152 5616 raft_consensus.cc:493] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:01.501209 5616 raft_consensus.cc:515] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.501446 5616 leader_election.cc:290] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.504057 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.501584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:42891 (local address 127.28.181.1:43519)
0201 03:09:01.501710 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.501720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.501805 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.501940 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.501941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.501942 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.502214 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.502216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.502969 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.502969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.503799 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.503800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.503882 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.503885 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.503916 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.503950 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260201 03:09:01.504153 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.501585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36313 (local address 127.28.181.1:44425)
0201 03:09:01.501715 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.501723 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.501805 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.501940 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.501941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.501942 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:01.502214 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.502216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.502969 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.502970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.503794 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.503796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.503881 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.503885 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.503916 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.503950 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260201 03:09:01.504341 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.501626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:43519 (local address 127.28.181.3:42891)
0201 03:09:01.501680 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.501682 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.501792 (+ 110us) server_negotiation.cc:408] Connection header received
0201 03:09:01.501853 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.501854 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.501863 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.501921 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.502281 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.502895 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.503875 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.504244 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.504250 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.504268 (+ 18us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.504290 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:09:01.504341 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.501630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:44425 (local address 127.28.181.2:36313)
0201 03:09:01.501680 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.501682 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.501792 (+ 110us) server_negotiation.cc:408] Connection header received
0201 03:09:01.501854 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.501855 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.501871 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.501921 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:01.502285 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.502904 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.503885 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.504244 (+ 359us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.504250 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.504268 (+ 18us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.504293 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:09:01.504596 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21db0dc4ae2a47f9ab39b1beacdd9bb3" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be" is_pre_election: true
I20260201 03:09:01.504585 5395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21db0dc4ae2a47f9ab39b1beacdd9bb3" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800170294934b47b54d0cb122820ec0" is_pre_election: true
I20260201 03:09:01.504683 5537 raft_consensus.cc:2468] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 0.
I20260201 03:09:01.504691 5395 raft_consensus.cc:2468] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 0.
I20260201 03:09:01.504827 5185 leader_election.cc:304] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.504945 5616 raft_consensus.cc:2804] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:01.504987 5616 raft_consensus.cc:493] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:01.505007 5616 raft_consensus.cc:3060] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.505483 5616 raft_consensus.cc:515] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.505599 5616 leader_election.cc:290] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 election: Requested vote from peers 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.505724 5395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21db0dc4ae2a47f9ab39b1beacdd9bb3" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800170294934b47b54d0cb122820ec0"
I20260201 03:09:01.505738 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21db0dc4ae2a47f9ab39b1beacdd9bb3" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be"
I20260201 03:09:01.505815 5537 raft_consensus.cc:3060] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.505827 5395 raft_consensus.cc:3060] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.506309 5537 raft_consensus.cc:2468] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 1.
I20260201 03:09:01.506399 5395 raft_consensus.cc:2468] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 1.
I20260201 03:09:01.506448 5185 leader_election.cc:304] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.506605 5616 raft_consensus.cc:2804] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:01.506786 5616 raft_consensus.cc:697] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 LEADER]: Becoming Leader. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Running, Role: LEADER
I20260201 03:09:01.506855 5616 consensus_queue.cc:237] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } }
I20260201 03:09:01.507159 5649 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: TxnStatusTablet state changed. Reason: New leader b561e78fb9904314a63f1f2d0c48c7bf. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.507236 5649 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:09:01.507398 5651 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:09:01.507349 5094 catalog_manager.cc:5654] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf reported cstate change: term changed from 0 to 1, leader changed from <none> to b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1). New cstate: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } health_report { overall_health: HEALTHY } } }
I20260201 03:09:01.507472 5651 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260201 03:09:01.526666 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.524677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.28.181.1:42315)
0201 03:09:01.524730 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.524732 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.524740 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:01.524829 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.524830 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.524844 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.524912 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.525184 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.525567 (+ 383us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.526268 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.526397 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.526420 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.526485 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.526532 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.526614 (+ 82us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.526630 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:09:01.526703 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.524622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:42315 (local address 127.0.0.1:51944)
0201 03:09:01.524689 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.524713 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.524758 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.524951 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.524952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.524954 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:09:01.525119 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.525120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.525637 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.525638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.526202 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.526204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.526271 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.526286 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.526550 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.526557 (+ 7us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.526602 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.526641 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260201 03:09:01.527750 5537 raft_consensus.cc:1275] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Refusing update from remote peer b561e78fb9904314a63f1f2d0c48c7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.527765 5395 raft_consensus.cc:1275] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Refusing update from remote peer b561e78fb9904314a63f1f2d0c48c7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.527992 5616 consensus_queue.cc:1048] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.528095 5649 consensus_queue.cc:1048] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.528554 5614 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: TxnStatusTablet state changed. Reason: New leader b561e78fb9904314a63f1f2d0c48c7bf. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.528623 5614 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.528648 5627 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: TxnStatusTablet state changed. Reason: New leader b561e78fb9904314a63f1f2d0c48c7bf. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.528703 5627 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.529263 5616 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.529330 5616 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:09:01.529376 5616 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.529419 5616 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P b561e78fb9904314a63f1f2d0c48c7bf: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:09:01.529884 5627 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.529954 5627 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 9800170294934b47b54d0cb122820ec0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.529975 5614 tablet_replica.cc:442] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } } peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } }
I20260201 03:09:01.530033 5614 tablet_replica.cc:445] T 21db0dc4ae2a47f9ab39b1beacdd9bb3 P 45293ded2bb2499b8c811f19f7d0d0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:09:01.534456 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.532493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:42315 (local address 127.28.181.2:60221)
0201 03:09:01.532653 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.532660 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.532730 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.532873 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.532873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.532875 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:09:01.532994 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.532995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.533694 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.533695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.534058 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.534060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.534115 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.534126 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.534359 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.534360 (+ 1us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.534391 (+ 31us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.534416 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260201 03:09:01.534477 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.532546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:60221 (local address 127.28.181.1:42315)
0201 03:09:01.532593 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.532595 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.532727 (+ 132us) server_negotiation.cc:408] Connection header received
0201 03:09:01.532796 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.532797 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.532806 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.532859 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.533052 (+ 193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.533635 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.534120 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.534263 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.534280 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.534300 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.534354 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.534408 (+ 54us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.534425 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:09:01.536041 5253 raft_consensus.cc:1275] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.536051 5537 raft_consensus.cc:1275] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.536216 5618 consensus_queue.cc:1048] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.536310 5627 consensus_queue.cc:1048] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.544308 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.542404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:40407 (local address 127.28.181.2:36313)
0201 03:09:01.542507 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.542508 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.542517 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:09:01.542534 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.542534 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.542548 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.542612 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.542907 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.543463 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.543950 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.544092 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.544111 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.544128 (+ 17us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.544185 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.544239 (+ 54us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.544256 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260201 03:09:01.544329 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.540892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36313 (local address 127.28.181.1:40407)
0201 03:09:01.541007 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.541016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:01.541062 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:01.542657 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:01.542658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:01.542659 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:09:01.542810 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.542811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.543528 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.543529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:01.543870 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:01.543871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.543951 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.543965 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.544187 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.544189 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:01.544233 (+ 44us) client_negotiation.cc:240] Negotiation successful
0201 03:09:01.544267 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260201 03:09:01.551966 5093 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.552024 5093 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.552501 5093 catalog_manager.cc:5941] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Sending DeleteTablet for 3 replicas of tablet 83c9f34afa254ee389581319af2b0c4e
I20260201 03:09:01.552716 5517 tablet_service.cc:1552] Processing DeleteTablet for tablet 83c9f34afa254ee389581319af2b0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260201 03:09:01.552729 5375 tablet_service.cc:1552] Processing DeleteTablet for tablet 83c9f34afa254ee389581319af2b0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:48540
I20260201 03:09:01.552798 5632 tablet_replica.cc:333] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: stopping tablet replica
I20260201 03:09:01.552814 5633 tablet_replica.cc:333] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: stopping tablet replica
I20260201 03:09:01.552834 5632 raft_consensus.cc:2243] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:09:01.552857 5633 raft_consensus.cc:2243] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.552887 5633 raft_consensus.cc:2272] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.552906 5632 raft_consensus.cc:2272] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.553256 5632 ts_tablet_manager.cc:1916] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.553278 5633 ts_tablet_manager.cc:1916] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.553951 5233 tablet_service.cc:1552] Processing DeleteTablet for tablet 83c9f34afa254ee389581319af2b0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:51928
I20260201 03:09:01.554589 5633 ts_tablet_manager.cc:1929] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:09:01.554637 5633 log.cc:1199] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/83c9f34afa254ee389581319af2b0c4e
I20260201 03:09:01.554812 5633 ts_tablet_manager.cc:1950] T 83c9f34afa254ee389581319af2b0c4e P 45293ded2bb2499b8c811f19f7d0d0be: Deleting consensus metadata
I20260201 03:09:01.554939 5631 tablet_replica.cc:333] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: stopping tablet replica
I20260201 03:09:01.554978 5631 raft_consensus.cc:2243] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.555006 5631 raft_consensus.cc:2272] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.556344 5632 ts_tablet_manager.cc:1929] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:09:01.556396 5632 log.cc:1199] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/83c9f34afa254ee389581319af2b0c4e
I20260201 03:09:01.556557 5631 ts_tablet_manager.cc:1916] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.556587 5632 ts_tablet_manager.cc:1950] T 83c9f34afa254ee389581319af2b0c4e P 9800170294934b47b54d0cb122820ec0: Deleting consensus metadata
I20260201 03:09:01.556579 5079 catalog_manager.cc:4985] TS 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891): tablet 83c9f34afa254ee389581319af2b0c4e (table test-table [id=b1c68586d13940d7b8a3170cf9bfb265]) successfully deleted
I20260201 03:09:01.556824 5078 catalog_manager.cc:4985] TS 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313): tablet 83c9f34afa254ee389581319af2b0c4e (table test-table [id=b1c68586d13940d7b8a3170cf9bfb265]) successfully deleted
I20260201 03:09:01.557853 5631 ts_tablet_manager.cc:1929] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:09:01.557909 5631 log.cc:1199] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/83c9f34afa254ee389581319af2b0c4e
I20260201 03:09:01.558113 5631 ts_tablet_manager.cc:1950] T 83c9f34afa254ee389581319af2b0c4e P b561e78fb9904314a63f1f2d0c48c7bf: Deleting consensus metadata
I20260201 03:09:01.558379 5080 catalog_manager.cc:4985] TS b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315): tablet 83c9f34afa254ee389581319af2b0c4e (table test-table [id=b1c68586d13940d7b8a3170cf9bfb265]) successfully deleted
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:09:01.576134 5143 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:01.571446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.28.181.62:42935)
0201 03:09:01.571508 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.571511 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.571664 (+ 153us) server_negotiation.cc:408] Connection header received
0201 03:09:01.571719 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.571720 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.571727 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.571767 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.572506 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.572937 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.573595 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.573685 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.576062 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54478: BlockingRecv error: recv got EOF from 127.0.0.1:54478 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM
I20260201 03:09:01.582262 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.576863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.28.181.62:42935)
0201 03:09:01.576897 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.576899 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.576959 (+ 60us) server_negotiation.cc:408] Connection header received
0201 03:09:01.576992 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.576993 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.577002 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.577061 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:01.577523 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.578163 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.578608 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.578750 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.580178 (+ 1428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:01.580181 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:01.580181 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:01.580187 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:01.581427 (+ 1240us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.581676 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.581677 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.581678 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.581699 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:01.581878 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:01.581878 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:01.581879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:01.581970 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:01.582054 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.582117 (+ 63us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.582187 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:09:01.583396 5093 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:09:01.583528 5093 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:01.584574 5375 tablet_service.cc:1505] Processing CreateTablet for tablet a06555a798f547fa88e638e9670e7bc5 (DEFAULT_TABLE table=test-table [id=e79c348ec07e421490e93aa3fbd5973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.584592 5233 tablet_service.cc:1505] Processing CreateTablet for tablet a06555a798f547fa88e638e9670e7bc5 (DEFAULT_TABLE table=test-table [id=e79c348ec07e421490e93aa3fbd5973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.584671 5233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06555a798f547fa88e638e9670e7bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.584671 5375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06555a798f547fa88e638e9670e7bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.585716 5609 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Bootstrap starting.
I20260201 03:09:01.585732 5517 tablet_service.cc:1505] Processing CreateTablet for tablet a06555a798f547fa88e638e9670e7bc5 (DEFAULT_TABLE table=test-table [id=e79c348ec07e421490e93aa3fbd5973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.585827 5611 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Bootstrap starting.
I20260201 03:09:01.585837 5517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06555a798f547fa88e638e9670e7bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.586030 5609 tablet_bootstrap.cc:654] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.586231 5611 tablet_bootstrap.cc:654] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.586933 5611 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: No bootstrap required, opened a new log
I20260201 03:09:01.586974 5611 ts_tablet_manager.cc:1403] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:09:01.587031 5608 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Bootstrap starting.
I20260201 03:09:01.587100 5611 raft_consensus.cc:359] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.587153 5611 raft_consensus.cc:385] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.587168 5611 raft_consensus.cc:740] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.587205 5611 consensus_queue.cc:260] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.587256 5611 ts_tablet_manager.cc:1434] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.587368 5608 tablet_bootstrap.cc:654] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.587906 5609 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: No bootstrap required, opened a new log
I20260201 03:09:01.587957 5609 ts_tablet_manager.cc:1403] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:01.588058 5609 raft_consensus.cc:359] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.588112 5609 raft_consensus.cc:385] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.588130 5609 raft_consensus.cc:740] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.588163 5609 consensus_queue.cc:260] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.588243 5609 ts_tablet_manager.cc:1434] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.588465 5608 tablet_bootstrap.cc:492] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: No bootstrap required, opened a new log
I20260201 03:09:01.588510 5608 ts_tablet_manager.cc:1403] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:01.588634 5608 raft_consensus.cc:359] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.588688 5608 raft_consensus.cc:385] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.588716 5608 raft_consensus.cc:740] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45293ded2bb2499b8c811f19f7d0d0be, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.588748 5608 consensus_queue.cc:260] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.588814 5608 ts_tablet_manager.cc:1434] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.685355 5649 raft_consensus.cc:493] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:01.685426 5649 raft_consensus.cc:515] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.685557 5649 leader_election.cc:290] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.685708 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a06555a798f547fa88e638e9670e7bc5" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be" is_pre_election: true
I20260201 03:09:01.685722 5395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a06555a798f547fa88e638e9670e7bc5" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800170294934b47b54d0cb122820ec0" is_pre_election: true
I20260201 03:09:01.685822 5537 raft_consensus.cc:2468] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 0.
I20260201 03:09:01.685827 5395 raft_consensus.cc:2468] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 0.
I20260201 03:09:01.685989 5185 leader_election.cc:304] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.686100 5649 raft_consensus.cc:2804] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:01.686149 5649 raft_consensus.cc:493] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:01.686164 5649 raft_consensus.cc:3060] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.686708 5649 raft_consensus.cc:515] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.686826 5649 leader_election.cc:290] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 election: Requested vote from peers 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.686980 5395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a06555a798f547fa88e638e9670e7bc5" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800170294934b47b54d0cb122820ec0"
I20260201 03:09:01.686990 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a06555a798f547fa88e638e9670e7bc5" candidate_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be"
I20260201 03:09:01.687037 5395 raft_consensus.cc:3060] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.687053 5537 raft_consensus.cc:3060] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.687553 5537 raft_consensus.cc:2468] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 1.
I20260201 03:09:01.687604 5395 raft_consensus.cc:2468] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b561e78fb9904314a63f1f2d0c48c7bf in term 1.
I20260201 03:09:01.687748 5185 leader_election.cc:304] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45293ded2bb2499b8c811f19f7d0d0be, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.687855 5649 raft_consensus.cc:2804] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:01.687927 5649 raft_consensus.cc:697] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 LEADER]: Becoming Leader. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Running, Role: LEADER
I20260201 03:09:01.687978 5649 consensus_queue.cc:237] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.688474 5093 catalog_manager.cc:5654] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf reported cstate change: term changed from 0 to 1, leader changed from <none> to b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1). New cstate: current_term: 1 leader_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } health_report { overall_health: UNKNOWN } } }
I20260201 03:09:01.702878 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.701404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.28.181.1:42315)
0201 03:09:01.701457 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.701459 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.701466 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:01.701483 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.701483 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.701491 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.701534 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.701709 (+ 175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.702129 (+ 420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.702583 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.702673 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.702686 (+ 13us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.702702 (+ 16us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.702738 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.702820 (+ 82us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.702837 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:09:01.703539 5395 raft_consensus.cc:1275] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Refusing update from remote peer b561e78fb9904314a63f1f2d0c48c7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.703539 5537 raft_consensus.cc:1275] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Refusing update from remote peer b561e78fb9904314a63f1f2d0c48c7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.703753 5649 consensus_queue.cc:1048] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.703879 5656 consensus_queue.cc:1048] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.706506 5093 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.706578 5093 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.707089 5093 catalog_manager.cc:5941] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Sending DeleteTablet for 3 replicas of tablet a06555a798f547fa88e638e9670e7bc5
I20260201 03:09:01.707276 5233 tablet_service.cc:1552] Processing DeleteTablet for tablet a06555a798f547fa88e638e9670e7bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:51928
I20260201 03:09:01.707381 5631 tablet_replica.cc:333] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: stopping tablet replica
I20260201 03:09:01.707427 5631 raft_consensus.cc:2243] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:09:01.707502 5631 raft_consensus.cc:2272] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.707504 5093 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:09:01.707608 5093 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:01.707769 5631 ts_tablet_manager.cc:1916] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.708541 5233 tablet_service.cc:1505] Processing CreateTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 (DEFAULT_TABLE table=test-table [id=1da5abb6d37545b1ac12765e2f8bc58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.708633 5233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.709125 5631 ts_tablet_manager.cc:1929] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.709175 5631 log.cc:1199] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-0/wal/wals/a06555a798f547fa88e638e9670e7bc5
I20260201 03:09:01.709363 5631 ts_tablet_manager.cc:1950] T a06555a798f547fa88e638e9670e7bc5 P b561e78fb9904314a63f1f2d0c48c7bf: Deleting consensus metadata
I20260201 03:09:01.709538 5611 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: Bootstrap starting.
I20260201 03:09:01.709595 5080 catalog_manager.cc:4985] TS b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315): tablet a06555a798f547fa88e638e9670e7bc5 (table test-table [id=e79c348ec07e421490e93aa3fbd5973c]) successfully deleted
I20260201 03:09:01.709820 5611 tablet_bootstrap.cc:654] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.709955 5375 tablet_service.cc:1552] Processing DeleteTablet for tablet a06555a798f547fa88e638e9670e7bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:48540
I20260201 03:09:01.710000 5517 tablet_service.cc:1552] Processing DeleteTablet for tablet a06555a798f547fa88e638e9670e7bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260201 03:09:01.710047 5374 tablet_service.cc:1505] Processing CreateTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 (DEFAULT_TABLE table=test-table [id=1da5abb6d37545b1ac12765e2f8bc58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.710125 5374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.711354 5609 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Bootstrap starting.
I20260201 03:09:01.711397 5516 tablet_service.cc:1505] Processing CreateTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 (DEFAULT_TABLE table=test-table [id=1da5abb6d37545b1ac12765e2f8bc58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:01.711484 5516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:01.711586 5609 tablet_bootstrap.cc:654] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.712059 5609 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: No bootstrap required, opened a new log
I20260201 03:09:01.712085 5609 ts_tablet_manager.cc:1403] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:09:01.712194 5609 raft_consensus.cc:359] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.712226 5609 raft_consensus.cc:385] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.712237 5609 raft_consensus.cc:740] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.712255 5609 consensus_queue.cc:260] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.712329 5609 ts_tablet_manager.cc:1434] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.712957 5608 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Bootstrap starting.
I20260201 03:09:01.712970 5632 tablet_replica.cc:333] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: stopping tablet replica
I20260201 03:09:01.713018 5632 raft_consensus.cc:2243] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.713054 5632 raft_consensus.cc:2272] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.713109 5633 tablet_replica.cc:333] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: stopping tablet replica
I20260201 03:09:01.713145 5633 raft_consensus.cc:2243] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.713181 5633 raft_consensus.cc:2272] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.713308 5608 tablet_bootstrap.cc:654] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:01.713728 5632 ts_tablet_manager.cc:1916] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.713835 5633 ts_tablet_manager.cc:1916] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.714061 5611 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: No bootstrap required, opened a new log
I20260201 03:09:01.714094 5611 ts_tablet_manager.cc:1403] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260201 03:09:01.714195 5611 raft_consensus.cc:359] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.714246 5611 raft_consensus.cc:385] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.714262 5611 raft_consensus.cc:740] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561e78fb9904314a63f1f2d0c48c7bf, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.714294 5611 consensus_queue.cc:260] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.714370 5611 ts_tablet_manager.cc:1434] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.715005 5632 ts_tablet_manager.cc:1929] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.715008 5633 ts_tablet_manager.cc:1929] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:01.715052 5632 log.cc:1199] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/a06555a798f547fa88e638e9670e7bc5
I20260201 03:09:01.715052 5633 log.cc:1199] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/a06555a798f547fa88e638e9670e7bc5
I20260201 03:09:01.715252 5632 ts_tablet_manager.cc:1950] T a06555a798f547fa88e638e9670e7bc5 P 9800170294934b47b54d0cb122820ec0: Deleting consensus metadata
I20260201 03:09:01.715252 5633 ts_tablet_manager.cc:1950] T a06555a798f547fa88e638e9670e7bc5 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting consensus metadata
I20260201 03:09:01.715561 5079 catalog_manager.cc:4985] TS 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891): tablet a06555a798f547fa88e638e9670e7bc5 (table test-table [id=e79c348ec07e421490e93aa3fbd5973c]) successfully deleted
I20260201 03:09:01.715667 5078 catalog_manager.cc:4985] TS 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313): tablet a06555a798f547fa88e638e9670e7bc5 (table test-table [id=e79c348ec07e421490e93aa3fbd5973c]) successfully deleted
I20260201 03:09:01.715687 5608 tablet_bootstrap.cc:492] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: No bootstrap required, opened a new log
I20260201 03:09:01.715720 5608 ts_tablet_manager.cc:1403] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260201 03:09:01.715840 5608 raft_consensus.cc:359] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.715898 5608 raft_consensus.cc:385] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:01.715909 5608 raft_consensus.cc:740] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45293ded2bb2499b8c811f19f7d0d0be, State: Initialized, Role: FOLLOWER
I20260201 03:09:01.715956 5608 consensus_queue.cc:260] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.716040 5608 ts_tablet_manager.cc:1434] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:01.784786 5615 raft_consensus.cc:493] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:01.784848 5615 raft_consensus.cc:515] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.784977 5615 leader_election.cc:290] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.785104 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f5c37c3f3e4ab9b653c20fa5f6b1a0" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be" is_pre_election: true
I20260201 03:09:01.785104 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f5c37c3f3e4ab9b653c20fa5f6b1a0" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" is_pre_election: true
I20260201 03:09:01.785173 5253 raft_consensus.cc:2468] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.785173 5537 raft_consensus.cc:2468] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 0.
I20260201 03:09:01.785288 5328 leader_election.cc:304] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9800170294934b47b54d0cb122820ec0, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.785406 5615 raft_consensus.cc:2804] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:01.785436 5615 raft_consensus.cc:493] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:01.785463 5615 raft_consensus.cc:3060] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.786103 5615 raft_consensus.cc:515] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.786233 5615 leader_election.cc:290] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Requested vote from peers b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315), 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891)
I20260201 03:09:01.786341 5253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f5c37c3f3e4ab9b653c20fa5f6b1a0" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b561e78fb9904314a63f1f2d0c48c7bf"
I20260201 03:09:01.786415 5253 raft_consensus.cc:3060] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.786417 5537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f5c37c3f3e4ab9b653c20fa5f6b1a0" candidate_uuid: "9800170294934b47b54d0cb122820ec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45293ded2bb2499b8c811f19f7d0d0be"
I20260201 03:09:01.786487 5537 raft_consensus.cc:3060] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:01.787071 5253 raft_consensus.cc:2468] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.787081 5537 raft_consensus.cc:2468] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800170294934b47b54d0cb122820ec0 in term 1.
I20260201 03:09:01.787233 5328 leader_election.cc:304] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9800170294934b47b54d0cb122820ec0, b561e78fb9904314a63f1f2d0c48c7bf; no voters:
I20260201 03:09:01.787320 5615 raft_consensus.cc:2804] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:01.787395 5615 raft_consensus.cc:697] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Becoming Leader. State: Replica: 9800170294934b47b54d0cb122820ec0, State: Running, Role: LEADER
I20260201 03:09:01.787436 5615 consensus_queue.cc:237] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } }
I20260201 03:09:01.787951 5093 catalog_manager.cc:5654] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9800170294934b47b54d0cb122820ec0 (127.28.181.2). New cstate: current_term: 1 leader_uuid: "9800170294934b47b54d0cb122820ec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9800170294934b47b54d0cb122820ec0" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 } health_report { overall_health: UNKNOWN } } }
I20260201 03:09:01.807922 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.806243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.28.181.2:36313)
0201 03:09:01.806280 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:01.806282 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:01.806289 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:01.806326 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:01.806327 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:01.806336 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:01.806372 (+ 36us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:01.806596 (+ 224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.807042 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:01.807530 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:01.807656 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:01.807670 (+ 14us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:01.807687 (+ 17us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:01.807745 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:01.807825 (+ 80us) server_negotiation.cc:300] Negotiation successful
0201 03:09:01.807846 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260201 03:09:01.809552 5537 raft_consensus.cc:1275] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.809619 5253 raft_consensus.cc:1275] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Refusing update from remote peer 9800170294934b47b54d0cb122820ec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:01.809741 5615 consensus_queue.cc:1048] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45293ded2bb2499b8c811f19f7d0d0be" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.809834 5627 consensus_queue.cc:1048] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b561e78fb9904314a63f1f2d0c48c7bf" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:01.819121 5094 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.819200 5094 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:01.819720 5094 catalog_manager.cc:5941] T 00000000000000000000000000000000 P aa4c8db1781449c3a8ee7fdb222185e3: Sending DeleteTablet for 3 replicas of tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0
I20260201 03:09:01.819911 5233 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:51928
I20260201 03:09:01.819926 5374 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:48540
I20260201 03:09:01.820014 5631 tablet_replica.cc:333] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf: stopping tablet replica
I20260201 03:09:01.820025 5632 tablet_replica.cc:333] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: stopping tablet replica
I20260201 03:09:01.820060 5632 raft_consensus.cc:2243] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:09:01.820060 5631 raft_consensus.cc:2243] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.820096 5631 raft_consensus.cc:2272] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P b561e78fb9904314a63f1f2d0c48c7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.820115 5632 raft_consensus.cc:2272] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.820163 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5168
I20260201 03:09:01.821380 5632 ts_tablet_manager.cc:1916] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.822803 5632 ts_tablet_manager.cc:1929] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:09:01.822846 5632 log.cc:1199] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-1/wal/wals/38f5c37c3f3e4ab9b653c20fa5f6b1a0
I20260201 03:09:01.822830 5516 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:01 UTC) from {username='oryx', principal='oryx/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260201 03:09:01.822934 5633 tablet_replica.cc:333] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: stopping tablet replica
I20260201 03:09:01.822973 5633 raft_consensus.cc:2243] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:01.823030 5633 raft_consensus.cc:2272] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:01.823067 5632 ts_tablet_manager.cc:1950] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 9800170294934b47b54d0cb122820ec0: Deleting consensus metadata
I20260201 03:09:01.823325 5078 catalog_manager.cc:4985] TS 9800170294934b47b54d0cb122820ec0 (127.28.181.2:36313): tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 (table test-table [id=1da5abb6d37545b1ac12765e2f8bc58f]) successfully deleted
W20260201 03:09:01.826926 5080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.181.1:42315 (error 108)
I20260201 03:09:01.827136 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5310
I20260201 03:09:01.827267 5633 ts_tablet_manager.cc:1916] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:01.828765 5633 ts_tablet_manager.cc:1929] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:09:01.828819 5633 log.cc:1199] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/38f5c37c3f3e4ab9b653c20fa5f6b1a0
I20260201 03:09:01.829000 5633 ts_tablet_manager.cc:1950] T 38f5c37c3f3e4ab9b653c20fa5f6b1a0 P 45293ded2bb2499b8c811f19f7d0d0be: Deleting consensus metadata
I20260201 03:09:01.829248 5079 catalog_manager.cc:4985] TS 45293ded2bb2499b8c811f19f7d0d0be (127.28.181.3:42891): tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0 (table test-table [id=1da5abb6d37545b1ac12765e2f8bc58f]) successfully deleted
I20260201 03:09:01.832278 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:01.827230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:42315 (local address 127.0.0.1:51970)
0201 03:09:01.832209 (+ 4979us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:01.832237 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4963,"thread_start_us":38,"threads_started":1}
W20260201 03:09:01.832381 5080 catalog_manager.cc:4712] TS b561e78fb9904314a63f1f2d0c48c7bf (127.28.181.1:42315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38f5c37c3f3e4ab9b653c20fa5f6b1a0: Network error: Client connection negotiation failed: client connection to 127.28.181.1:42315: connect: Connection refused (error 111)
I20260201 03:09:01.832464 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5452
I20260201 03:09:01.836593 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5062
2026-02-01T03:09:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3130 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5694](info): set up 2 sockets
Feb 01 03:09:01 dist-test-slave-ck1n krb5kdc[5694](info): commencing operation
krb5kdc: starting...
W20260201 03:09:03.546083 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.690s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:09:03 dist-test-slave-ck1n krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:09:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:03.652191 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:41389
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:41389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:03.718982 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:03.719151 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:03.719178 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:03.720520 5710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:03.720583 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:03.720607 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:03.720635 5710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:03.720665 5710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:03.722132 5710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:41389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:41389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:03.722366 5710 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:03.722532 5710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:09:03.724644 5710 server_base.cc:1060] running on GCE node
W20260201 03:09:03.724663 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:03.724671 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:03.724802 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:03.725000 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:03.725209 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:03.726343 5710 hybrid_clock.cc:648] HybridClock initialized: now 1769915343726328 us; error 30 us; skew 500 ppm
W20260201 03:09:03.727715 5710 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-02-01T03:09:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1891 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:09:03 dist-test-slave-ck1n krb5kdc[5723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:09:03 dist-test-slave-ck1n krb5kdc[5723](info): set up 2 sockets
Feb 01 03:09:03 dist-test-slave-ck1n krb5kdc[5723](info): commencing operation
krb5kdc: starting...
W20260201 03:09:05.446096 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.698s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:09:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:05.553575 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38983
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:05.619987 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:05.620157 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:05.620187 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:05.621368 5739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:05.621428 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:05.621447 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:05.621464 5739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:05.621491 5739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:05.622802 5739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:05.622998 5739 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:05.623167 5739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:05.625216 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:05.625212 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:05.625249 5739 server_base.cc:1060] running on GCE node
W20260201 03:09:05.625212 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:05.625537 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:05.625733 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:05.626911 5739 hybrid_clock.cc:648] HybridClock initialized: now 1769915345626905 us; error 25 us; skew 500 ppm
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:05.628732 5739 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:09:05.629232 5739 webserver.cc:492] Webserver started at http://127.28.181.62:41011/ using document root <none> and password file <none>
I20260201 03:09:05.629395 5739 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:05.629441 5739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:05.629554 5739 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:05.630388 5739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "a4cff5fe13124610b19d1afb8ed392b4"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "1e2f410b2961b7e466bea4f70e2063cd"
server_key_iv: "13c0cba8a80111fffc06e0cab5249fbb"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.630653 5739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "a4cff5fe13124610b19d1afb8ed392b4"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "1e2f410b2961b7e466bea4f70e2063cd"
server_key_iv: "13c0cba8a80111fffc06e0cab5249fbb"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.631639 5739 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:05.632174 5754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.632316 5739 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:05.632391 5739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "a4cff5fe13124610b19d1afb8ed392b4"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "1e2f410b2961b7e466bea4f70e2063cd"
server_key_iv: "13c0cba8a80111fffc06e0cab5249fbb"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.632455 5739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:05.638473 5739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:05.639751 5739 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:05.639873 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:05.643251 5739 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:38983
I20260201 03:09:05.643273 5806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:38983 every 8 connection(s)
I20260201 03:09:05.643522 5739 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:05.643988 5807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:05.645908 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Bootstrap starting.
I20260201 03:09:05.646333 5807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:05.646504 5807 log.cc:826] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:05.646750 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5739
I20260201 03:09:05.646826 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:09:05.646978 29396 external_mini_cluster.cc:1468] Setting key 34056b21034b9dce4c948edd240a49e7
I20260201 03:09:05.647110 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: No bootstrap required, opened a new log
I20260201 03:09:05.648542 5807 raft_consensus.cc:359] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:05.648636 5807 raft_consensus.cc:385] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:05.648653 5807 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cff5fe13124610b19d1afb8ed392b4, State: Initialized, Role: FOLLOWER
I20260201 03:09:05.648752 5807 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:05.648808 5807 raft_consensus.cc:399] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:05.648833 5807 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:05.648880 5807 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:05.649362 5807 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:05.649461 5807 leader_election.cc:304] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cff5fe13124610b19d1afb8ed392b4; no voters:
I20260201 03:09:05.649621 5807 leader_election.cc:290] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:09:05.649650 5812 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:05.649817 5812 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 LEADER]: Becoming Leader. State: Replica: a4cff5fe13124610b19d1afb8ed392b4, State: Running, Role: LEADER
I20260201 03:09:05.649880 5807 sys_catalog.cc:565] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:05.649951 5812 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:05.650303 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4cff5fe13124610b19d1afb8ed392b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } } }
I20260201 03:09:05.650372 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:05.650561 5814 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cff5fe13124610b19d1afb8ed392b4. Latest consensus state: current_term: 1 leader_uuid: "a4cff5fe13124610b19d1afb8ed392b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } } }
I20260201 03:09:05.650677 5814 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:05.650803 5819 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:05.651229 5819 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:05.652853 5819 catalog_manager.cc:1357] Generated new cluster ID: 10d3a0ed31e1477b930b6e064e15c985
I20260201 03:09:05.652935 5819 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:09:05.655716 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:05.647331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.28.181.62:38983)
0201 03:09:05.647489 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:05.647499 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:05.647520 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:09:05.647991 (+ 471us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:05.647997 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:05.648048 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:05.648163 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:05.648846 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.649590 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:05.650242 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.650465 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:05.652587 (+ 2122us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:05.652592 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:05.652592 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:05.652605 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:05.654257 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.654572 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.654574 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.654575 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.654635 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.654775 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.654776 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.654776 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.654977 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:05.655092 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:05.655478 (+ 386us) server_negotiation.cc:300] Negotiation successful
0201 03:09:05.655577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260201 03:09:05.657110 5819 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:09:05.657516 5819 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:05.663266 5819 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Generated new TSK 0
I20260201 03:09:05.663400 5819 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:05.703649 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38983
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:05.769357 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:05.769517 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:05.769547 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:05.770748 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:05.770807 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:05.770844 5835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:05.772148 5835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:05.772356 5835 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:05.772521 5835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:05.774693 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:05.774786 5835 server_base.cc:1060] running on GCE node
W20260201 03:09:05.774706 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:05.774703 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:05.775074 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:05.775285 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:05.776414 5835 hybrid_clock.cc:648] HybridClock initialized: now 1769915345776401 us; error 30 us; skew 500 ppm
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:05.778564 5835 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:09:05.779110 5835 webserver.cc:492] Webserver started at http://127.28.181.1:37059/ using document root <none> and password file <none>
I20260201 03:09:05.779281 5835 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:05.779327 5835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:05.779418 5835 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:05.780117 5835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "7ddd31d0f3e54406bd456bcfcafee9bd"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "c6594f17b2ee1d8df40634086d9299c6"
server_key_iv: "ef8a5d084104c18a7fc5e9d5dc88dfd5"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.780339 5835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "7ddd31d0f3e54406bd456bcfcafee9bd"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "c6594f17b2ee1d8df40634086d9299c6"
server_key_iv: "ef8a5d084104c18a7fc5e9d5dc88dfd5"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.781242 5835 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:05.781749 5850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.781929 5835 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:05.781983 5835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "7ddd31d0f3e54406bd456bcfcafee9bd"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "c6594f17b2ee1d8df40634086d9299c6"
server_key_iv: "ef8a5d084104c18a7fc5e9d5dc88dfd5"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.782053 5835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:05.794530 5835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:05.796092 5835 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:05.796309 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:05.796492 5835 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:05.796759 5835 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:05.796787 5835 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.796803 5835 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:05.796814 5835 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.801553 5835 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:37029
I20260201 03:09:05.801581 5963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:37029 every 8 connection(s)
I20260201 03:09:05.801903 5835 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:05.806725 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5835
I20260201 03:09:05.806795 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:09:05.806955 29396 external_mini_cluster.cc:1468] Setting key ec73653d98c437a7de2c1e2247b8b3ec
I20260201 03:09:05.809233 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:05.802529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:51719 (local address 127.28.181.62:38983)
0201 03:09:05.802595 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:05.802597 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:05.802778 (+ 181us) server_negotiation.cc:408] Connection header received
0201 03:09:05.803242 (+ 464us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:05.803243 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:05.803253 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:05.803300 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:05.803896 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.804323 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:05.804798 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.804943 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:05.806865 (+ 1922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:05.806869 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:05.806869 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:05.806876 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:05.808243 (+ 1367us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.808538 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.808539 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.808540 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.808560 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.808701 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.808701 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.808702 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.808799 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:05.808858 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:05.809116 (+ 258us) server_negotiation.cc:300] Negotiation successful
0201 03:09:05.809177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:09:05.809422 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:05.802502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:51719)
0201 03:09:05.802645 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:05.802687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:05.803148 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:05.803347 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:05.803354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:05.803587 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:05.803804 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:05.803807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.804403 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:05.804403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:05.804722 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:05.804723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.804847 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:05.805583 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:05.805596 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:05.806750 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:05.808318 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:05.808323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:05.808324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:05.808480 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:05.808612 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:05.808613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:05.808613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:05.808655 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:05.808860 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:05.808865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:05.808994 (+ 129us) client_negotiation.cc:769] Sending connection context
0201 03:09:05.809101 (+ 107us) client_negotiation.cc:240] Negotiation successful
0201 03:09:05.809231 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260201 03:09:05.809867 5964 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38983
I20260201 03:09:05.809949 5964 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:05.810129 5964 heartbeater.cc:507] Master 127.28.181.62:38983 requested a full tablet report, sending...
I20260201 03:09:05.810536 5771 ts_manager.cc:194] Registered new tserver with Master: 7ddd31d0f3e54406bd456bcfcafee9bd (127.28.181.1:37029)
I20260201 03:09:05.811080 5771 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:51719
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:05.843482 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38983
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:05.908919 5971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:05.909055 5971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:05.909075 5971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:05.910214 5971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:05.910247 5971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:05.910269 5971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:05.911521 5971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:05.911657 5971 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:05.911764 5971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:05.913681 5977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:05.913687 5976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:05.913698 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:05.913918 5971 server_base.cc:1060] running on GCE node
I20260201 03:09:05.914065 5971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:05.914243 5971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:05.915376 5971 hybrid_clock.cc:648] HybridClock initialized: now 1769915345915362 us; error 28 us; skew 500 ppm
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:05.917312 5971 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:09:05.917820 5971 webserver.cc:492] Webserver started at http://127.28.181.2:39405/ using document root <none> and password file <none>
I20260201 03:09:05.917989 5971 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:05.918038 5971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:05.918169 5971 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:05.918980 5971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "e444c8f4d8ee4bbfb733b8f7162a444f"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "37ddc283006fb11521f0a3e8f902872d"
server_key_iv: "800be958730ab66c3aa310e18a20f9f4"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.919268 5971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "e444c8f4d8ee4bbfb733b8f7162a444f"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "37ddc283006fb11521f0a3e8f902872d"
server_key_iv: "800be958730ab66c3aa310e18a20f9f4"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.920251 5971 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:05.920794 5986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.920934 5971 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:05.921015 5971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "e444c8f4d8ee4bbfb733b8f7162a444f"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "37ddc283006fb11521f0a3e8f902872d"
server_key_iv: "800be958730ab66c3aa310e18a20f9f4"
server_key_version: "encryptionkey@0"
I20260201 03:09:05.921079 5971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:05.935103 5971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:05.936754 5971 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:05.936878 5971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:05.937062 5971 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:05.937319 5971 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:05.937366 5971 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.937402 5971 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:05.937422 5971 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:05.942283 5971 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:37751
I20260201 03:09:05.942317 6099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:37751 every 8 connection(s)
I20260201 03:09:05.942556 5971 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:09:05.946529 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 5971
I20260201 03:09:05.946599 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:09:05.946743 29396 external_mini_cluster.cc:1468] Setting key 1df7e8a92a459b3f0bda89c2d328ad07
Feb 01 03:09:05 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:05.950263 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:05.943133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:41471 (local address 127.28.181.62:38983)
0201 03:09:05.943195 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:05.943197 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:05.943315 (+ 118us) server_negotiation.cc:408] Connection header received
0201 03:09:05.943793 (+ 478us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:05.943794 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:05.943806 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:05.943882 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:05.944733 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.945152 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:05.945601 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.945690 (+ 89us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:05.947848 (+ 2158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:05.947850 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:05.947850 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:05.947856 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:05.949132 (+ 1276us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.949526 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.949527 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.949527 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.949543 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:05.949707 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:05.949707 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:05.949708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:05.949813 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:05.949865 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:05.950141 (+ 276us) server_negotiation.cc:300] Negotiation successful
0201 03:09:05.950202 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:09:05.950428 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:05.943095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:41471)
0201 03:09:05.943230 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:05.943251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:05.943667 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:05.943934 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:05.943938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:05.944325 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:05.944664 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:05.944668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.945224 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:05.945224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:05.945544 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:05.945545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:05.945627 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:05.946342 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:05.946355 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:05.947776 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:05.949197 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:05.949202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:05.949204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:05.949461 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:05.949593 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:05.949594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:05.949594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:05.949660 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:05.949871 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:05.949875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:05.950006 (+ 131us) client_negotiation.cc:769] Sending connection context
0201 03:09:05.950114 (+ 108us) client_negotiation.cc:240] Negotiation successful
0201 03:09:05.950237 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260201 03:09:05.950825 6100 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38983
I20260201 03:09:05.950919 6100 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:05.951076 6100 heartbeater.cc:507] Master 127.28.181.62:38983 requested a full tablet report, sending...
I20260201 03:09:05.951426 5771 ts_manager.cc:194] Registered new tserver with Master: e444c8f4d8ee4bbfb733b8f7162a444f (127.28.181.2:37751)
I20260201 03:09:05.951815 5771 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:41471
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:05.982743 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38983
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:06.049849 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:06.049981 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:06.050001 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:06.051179 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:06.051218 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:06.051240 6107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:06.052574 6107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:06.052723 6107 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:06.052834 6107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:06.054795 6115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.054807 6113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.054795 6112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.055078 6107 server_base.cc:1060] running on GCE node
I20260201 03:09:06.055215 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:06.055399 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:06.056515 6107 hybrid_clock.cc:648] HybridClock initialized: now 1769915346056500 us; error 33 us; skew 500 ppm
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:06.058517 6107 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:09:06.058959 6107 webserver.cc:492] Webserver started at http://127.28.181.3:32809/ using document root <none> and password file <none>
I20260201 03:09:06.059147 6107 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:06.059194 6107 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:06.059298 6107 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:06.060048 6107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "4ddb43cc95c64aeea047c9800dae22e5"
format_stamp: "Formatted at 2026-02-01 03:09:06 on dist-test-slave-ck1n"
server_key: "fd24ed4197a97b71ed6de8c53f66bd37"
server_key_iv: "c6724e2991e61912405a78e49aab4524"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.060313 6107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "4ddb43cc95c64aeea047c9800dae22e5"
format_stamp: "Formatted at 2026-02-01 03:09:06 on dist-test-slave-ck1n"
server_key: "fd24ed4197a97b71ed6de8c53f66bd37"
server_key_iv: "c6724e2991e61912405a78e49aab4524"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.061266 6107 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:06.061825 6122 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.061975 6107 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:06.062047 6107 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "4ddb43cc95c64aeea047c9800dae22e5"
format_stamp: "Formatted at 2026-02-01 03:09:06 on dist-test-slave-ck1n"
server_key: "fd24ed4197a97b71ed6de8c53f66bd37"
server_key_iv: "c6724e2991e61912405a78e49aab4524"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.062114 6107 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:06.088289 6107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:06.089666 6107 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:06.089838 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:06.090034 6107 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:06.090301 6107 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:06.090349 6107 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.090389 6107 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:06.090409 6107 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.095016 6107 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:36823
I20260201 03:09:06.095077 6235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:36823 every 8 connection(s)
I20260201 03:09:06.095333 6107 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:09:06.095865 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6107
I20260201 03:09:06.095976 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:09:06.096169 29396 external_mini_cluster.cc:1468] Setting key d70ec76bbd83515bc747c2ef154c971d
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:06.102340 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.096106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:41831 (local address 127.28.181.62:38983)
0201 03:09:06.096157 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.096159 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.096171 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:09:06.096643 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.096644 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.096654 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.096692 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.097386 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.097854 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.098361 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.098464 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.100312 (+ 1848us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:06.100315 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:06.100315 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:06.100320 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:06.101473 (+ 1153us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:06.101754 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:06.101755 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:06.101755 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:06.101811 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:06.101958 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:06.101958 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:06.101958 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:06.102024 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:06.102073 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:06.102234 (+ 161us) server_negotiation.cc:300] Negotiation successful
0201 03:09:06.102285 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260201 03:09:06.102439 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.095923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:41831)
0201 03:09:06.096030 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.096057 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.096575 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.096751 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.096755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.097071 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.097299 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.097301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.097921 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.097922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.098294 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.098295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.098387 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.099093 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.099106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.100249 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:06.101550 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:06.101553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:06.101555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:06.101701 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:06.101863 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:06.101863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:06.101864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:06.101914 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:06.102077 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:06.102079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:06.102159 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:09:06.102226 (+ 67us) client_negotiation.cc:240] Negotiation successful
0201 03:09:06.102302 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":58,"threads_started":1}
I20260201 03:09:06.102793 6236 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38983
I20260201 03:09:06.102875 6236 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:06.103051 6236 heartbeater.cc:507] Master 127.28.181.62:38983 requested a full tablet report, sending...
I20260201 03:09:06.103338 5771 ts_manager.cc:194] Registered new tserver with Master: 4ddb43cc95c64aeea047c9800dae22e5 (127.28.181.3:36823)
I20260201 03:09:06.103694 5771 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:41831
I20260201 03:09:06.107695 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.1.keytab.
Entry for principal oryx/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.1.keytab.
WARNING: no policy specified for oryx/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.2.keytab.
Entry for principal oryx/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.2.keytab.
WARNING: no policy specified for oryx/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.3.keytab.
Entry for principal oryx/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.3.keytab.
I20260201 03:09:06.158363 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5835
I20260201 03:09:06.162454 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5971
I20260201 03:09:06.166213 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6107
I20260201 03:09:06.169588 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 5739
I20260201 03:09:06.173956 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38983
--webserver_interface=127.28.181.62
--webserver_port=41011
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:06.239741 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:06.239881 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:06.239898 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:06.241029 6245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:06.241060 6245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:06.241072 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:06.241083 6245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:06.241093 6245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:06.242383 6245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=41011
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:06.242512 6245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:06.242620 6245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:06.244493 6253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.244529 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.244627 6245 server_base.cc:1060] running on GCE node
W20260201 03:09:06.244496 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.244832 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:06.244999 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:06.246137 6245 hybrid_clock.cc:648] HybridClock initialized: now 1769915346246121 us; error 29 us; skew 500 ppm
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:06.247998 6245 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:09:06.248487 6245 webserver.cc:492] Webserver started at http://127.28.181.62:41011/ using document root <none> and password file <none>
I20260201 03:09:06.248656 6245 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:06.248728 6245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:06.249711 6245 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:06.250245 6260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.250393 6245 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.250466 6245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "a4cff5fe13124610b19d1afb8ed392b4"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "1e2f410b2961b7e466bea4f70e2063cd"
server_key_iv: "13c0cba8a80111fffc06e0cab5249fbb"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.250689 6245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:06.270184 6245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:06.271812 6245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:06.271940 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:06.275297 6245 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:38983
I20260201 03:09:06.275328 6312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:38983 every 8 connection(s)
I20260201 03:09:06.275588 6245 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:06.276463 6313 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:09:06.276925 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6245
I20260201 03:09:06.276940 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Bootstrap starting.
I20260201 03:09:06.277158 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:37029
--local_ip_for_outbound_sockets=127.28.181.1
--tserver_master_addrs=127.28.181.62:38983
--webserver_port=37059
--webserver_interface=127.28.181.1
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.181.1
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:09:06.284004 6313 log.cc:826] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:06.285315 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:09:06.285552 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Bootstrap complete.
I20260201 03:09:06.287237 6313 raft_consensus.cc:359] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:06.287385 6313 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cff5fe13124610b19d1afb8ed392b4, State: Initialized, Role: FOLLOWER
I20260201 03:09:06.287501 6313 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:06.287559 6313 raft_consensus.cc:399] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:06.287606 6313 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:06.287647 6313 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 1 FOLLOWER]: Advancing to term 2
I20260201 03:09:06.288331 6313 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:06.288403 6313 leader_election.cc:304] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cff5fe13124610b19d1afb8ed392b4; no voters:
I20260201 03:09:06.288552 6313 leader_election.cc:290] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260201 03:09:06.288589 6317 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260201 03:09:06.288707 6317 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [term 2 LEADER]: Becoming Leader. State: Replica: a4cff5fe13124610b19d1afb8ed392b4, State: Running, Role: LEADER
I20260201 03:09:06.288754 6313 sys_catalog.cc:565] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:06.288838 6317 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } }
I20260201 03:09:06.289498 6319 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a4cff5fe13124610b19d1afb8ed392b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } } }
I20260201 03:09:06.289513 6318 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cff5fe13124610b19d1afb8ed392b4. Latest consensus state: current_term: 2 leader_uuid: "a4cff5fe13124610b19d1afb8ed392b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cff5fe13124610b19d1afb8ed392b4" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38983 } } }
I20260201 03:09:06.289649 6319 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:06.289657 6318 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:06.290092 6327 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:06.290601 6327 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:09:06.290710 6327 catalog_manager.cc:1269] Loaded cluster ID: 10d3a0ed31e1477b930b6e064e15c985
I20260201 03:09:06.290740 6327 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:09:06.291777 6327 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:06.291923 6327 catalog_manager.cc:6038] T 00000000000000000000000000000000 P a4cff5fe13124610b19d1afb8ed392b4: Loaded TSK: 0
I20260201 03:09:06.292088 6327 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260201 03:09:06.347106 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:06.347252 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:06.347285 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:06.348534 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:06.348589 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:06.348629 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:06.350015 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.1.keytab
--principal=oryx/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:37029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=37059
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:06.350214 6315 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:06.350363 6315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:06.352312 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.352500 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.352344 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.352814 6315 server_base.cc:1060] running on GCE node
I20260201 03:09:06.352958 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:06.353133 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:06.354269 6315 hybrid_clock.cc:648] HybridClock initialized: now 1769915346354235 us; error 50 us; skew 500 ppm
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:06.356272 6315 init.cc:377] Logged in from keytab as oryx/127.28.181.1@KRBTEST.COM (short username oryx)
I20260201 03:09:06.356686 6315 webserver.cc:492] Webserver started at http://127.28.181.1:37059/ using document root <none> and password file <none>
I20260201 03:09:06.356844 6315 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:06.356901 6315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:06.357937 6315 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:06.358439 6349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.358595 6315 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.358664 6315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "7ddd31d0f3e54406bd456bcfcafee9bd"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "c6594f17b2ee1d8df40634086d9299c6"
server_key_iv: "ef8a5d084104c18a7fc5e9d5dc88dfd5"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.358866 6315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:06.370205 6315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:06.371660 6315 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:06.371781 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:06.371956 6315 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:06.372206 6315 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:06.372258 6315 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.372293 6315 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:06.372313 6315 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.377575 6315 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:37029
I20260201 03:09:06.377631 6462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:37029 every 8 connection(s)
I20260201 03:09:06.377929 6315 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.383951 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6315
I20260201 03:09:06.384241 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:37751
--local_ip_for_outbound_sockets=127.28.181.2
--tserver_master_addrs=127.28.181.62:38983
--webserver_port=39405
--webserver_interface=127.28.181.2
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.181.2
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:09:06.384581 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.378496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:44249)
0201 03:09:06.378617 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.378670 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.379072 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.379460 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.379464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.379784 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.380116 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.380120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.380946 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.380947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.381451 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.381453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.381570 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.382355 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.382371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.384438 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
W20260201 03:09:06.384900 6463 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:38983 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.181.62:38983: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
W20260201 03:09:06.385748 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.378569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:44249 (local address 127.28.181.62:38983)
0201 03:09:06.379191 (+ 622us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.379198 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.379213 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:09:06.379281 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.379284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.379312 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.379385 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.380211 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.380854 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.381525 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.381659 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.385622 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:44249: BlockingRecv error: recv got EOF from 127.28.181.1:44249 (error 108)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":103,"threads_started":1}
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.388614 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.385111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:40563)
0201 03:09:06.385160 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.385171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.385209 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.385443 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.385444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.385568 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.385729 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.385730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.386405 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.386406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.386713 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.386714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.386793 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.387135 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.387138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.388554 (+ 1416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260201 03:09:06.388677 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.385203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:40563 (local address 127.28.181.62:38983)
0201 03:09:06.385319 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.385321 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.385332 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:09:06.385352 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.385353 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.385363 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.385813 (+ 450us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.385852 (+ 39us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.386327 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.386794 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.387224 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.388614 (+ 1390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:40563: BlockingRecv error: recv got EOF from 127.28.181.1:40563 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.392097 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.388809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:59987)
0201 03:09:06.388885 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.388893 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.388926 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.389119 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.389120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.389225 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.389311 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.389312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.389953 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.389954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.390290 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.390291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.390376 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.390706 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.390710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.392037 (+ 1327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260201 03:09:06.392141 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.388867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:59987 (local address 127.28.181.62:38983)
0201 03:09:06.389011 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.389013 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.389018 (+ 5us) server_negotiation.cc:408] Connection header received
0201 03:09:06.389035 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.389036 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.389043 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.389105 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.389388 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.389879 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.390376 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.390785 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.392094 (+ 1309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:59987: BlockingRecv error: recv got EOF from 127.28.181.1:59987 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260201 03:09:06.392364 6463 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260201 03:09:06.464557 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:06.464679 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:06.464697 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:06.465806 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:06.465837 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:06.465857 6468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:06.467092 6468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.2.keytab
--principal=oryx/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:37751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=39405
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:06.467217 6468 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:06.467322 6468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:09:06.469291 6468 server_base.cc:1060] running on GCE node
W20260201 03:09:06.469265 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.469249 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.469260 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.469583 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:06.469763 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:06.470922 6468 hybrid_clock.cc:648] HybridClock initialized: now 1769915346470919 us; error 31 us; skew 500 ppm
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:06.472764 6468 init.cc:377] Logged in from keytab as oryx/127.28.181.2@KRBTEST.COM (short username oryx)
I20260201 03:09:06.473232 6468 webserver.cc:492] Webserver started at http://127.28.181.2:39405/ using document root <none> and password file <none>
I20260201 03:09:06.473388 6468 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:06.473446 6468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:06.474459 6468 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.474897 6483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.475046 6468 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.475121 6468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "e444c8f4d8ee4bbfb733b8f7162a444f"
format_stamp: "Formatted at 2026-02-01 03:09:05 on dist-test-slave-ck1n"
server_key: "37ddc283006fb11521f0a3e8f902872d"
server_key_iv: "800be958730ab66c3aa310e18a20f9f4"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.475315 6468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:06.497162 6468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:06.498456 6468 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:06.498575 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:06.498745 6468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:06.499029 6468 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:06.499076 6468 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.499109 6468 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:06.499135 6468 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.503705 6468 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:37751
I20260201 03:09:06.503736 6596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:37751 every 8 connection(s)
I20260201 03:09:06.503980 6468 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.508697 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6468
I20260201 03:09:06.508926 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:36823
--local_ip_for_outbound_sockets=127.28.181.3
--tserver_master_addrs=127.28.181.62:38983
--webserver_port=32809
--webserver_interface=127.28.181.3
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.181.3
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:09:06.509299 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.504458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:53281)
0201 03:09:06.504627 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.504647 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.505018 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.505202 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.505209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.505433 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.505644 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.505647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.506281 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.506282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.506601 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.506603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.506717 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.507381 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.507394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.509215 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
W20260201 03:09:06.509397 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.504505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:53281 (local address 127.28.181.62:38983)
0201 03:09:06.504578 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.504580 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.504712 (+ 132us) server_negotiation.cc:408] Connection header received
0201 03:09:06.505102 (+ 390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.505103 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.505114 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.505158 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.505737 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.506194 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.506680 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.506841 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.509323 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:53281: BlockingRecv error: recv got EOF from 127.28.181.2:53281 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260201 03:09:06.509503 6597 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:38983 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.181.62:38983: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.512806 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.509683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:54813)
0201 03:09:06.509726 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.509735 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.509803 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.509921 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.509922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.510029 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.510134 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.510135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.510652 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.510653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.510941 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.510942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.511008 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.511372 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.511375 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.512746 (+ 1371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260201 03:09:06.512866 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.509741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:54813 (local address 127.28.181.62:38983)
0201 03:09:06.509799 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.509801 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.509808 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:06.509856 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.509857 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.509865 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.509918 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.510208 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.510586 (+ 378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.511016 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.511160 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.512811 (+ 1651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:54813: BlockingRecv error: recv got EOF from 127.28.181.2:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.516065 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.513022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:57581)
0201 03:09:06.513133 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.513145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.513196 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.513314 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.513314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.513419 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.513504 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.513505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.514083 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.514084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.514357 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.514358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.514449 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.514796 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.514799 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.516004 (+ 1205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260201 03:09:06.516139 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.513052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:57581 (local address 127.28.181.62:38983)
0201 03:09:06.513080 (+ 28us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.513082 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.513186 (+ 104us) server_negotiation.cc:408] Connection header received
0201 03:09:06.513244 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.513245 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.513254 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.513310 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.513574 (+ 264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.514018 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.514450 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.514588 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.516061 (+ 1473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:57581: BlockingRecv error: recv got EOF from 127.28.181.2:57581 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260201 03:09:06.516227 6597 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260201 03:09:06.576329 6600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:06.576453 6600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:06.576472 6600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:06.577636 6600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:06.577672 6600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:06.577694 6600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:06.579138 6600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/krb5kdc/oryx_127.28.181.3.keytab
--principal=oryx/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:36823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=32809
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:06.579324 6600 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:06.579478 6600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:06.581436 6606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.581436 6609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:06.581465 6607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:06.581692 6600 server_base.cc:1060] running on GCE node
I20260201 03:09:06.581832 6600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:06.581980 6600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:06.583093 6600 hybrid_clock.cc:648] HybridClock initialized: now 1769915346583078 us; error 28 us; skew 500 ppm
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915346, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:06.584954 6600 init.cc:377] Logged in from keytab as oryx/127.28.181.3@KRBTEST.COM (short username oryx)
I20260201 03:09:06.585332 6600 webserver.cc:492] Webserver started at http://127.28.181.3:32809/ using document root <none> and password file <none>
I20260201 03:09:06.585449 6600 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:06.585484 6600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:06.586491 6600 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:06.586900 6616 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.587083 6600 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.587185 6600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "4ddb43cc95c64aeea047c9800dae22e5"
format_stamp: "Formatted at 2026-02-01 03:09:06 on dist-test-slave-ck1n"
server_key: "fd24ed4197a97b71ed6de8c53f66bd37"
server_key_iv: "c6724e2991e61912405a78e49aab4524"
server_key_version: "encryptionkey@0"
I20260201 03:09:06.587374 6600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:06.599617 6600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:06.600765 6600 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:06.600881 6600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:06.601044 6600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:06.601279 6600 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:06.601322 6600 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.601356 6600 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:06.601377 6600 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:06.606400 6600 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:36823
I20260201 03:09:06.606465 6729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:36823 every 8 connection(s)
I20260201 03:09:06.606745 6600 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.611773 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.607323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:35867)
0201 03:09:06.607534 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.607554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.607886 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.608053 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.608056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.608274 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.608483 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.608485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.608997 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.608997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.609312 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.609313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.609391 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.610088 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.610100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.611687 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
W20260201 03:09:06.611811 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.607401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:35867 (local address 127.28.181.62:38983)
0201 03:09:06.607473 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.607475 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.607629 (+ 154us) server_negotiation.cc:408] Connection header received
0201 03:09:06.607948 (+ 319us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.607950 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.607958 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.608003 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.608545 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.608936 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.609370 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.609459 (+ 89us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.611756 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:35867: BlockingRecv error: recv got EOF from 127.28.181.3:35867 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260201 03:09:06.611953 6730 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:38983 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.181.62:38983: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
I20260201 03:09:06.612691 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6600
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.615595 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.612114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:40227)
0201 03:09:06.612158 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.612166 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.612197 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.612384 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.612384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.612483 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.612588 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.612589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.613267 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.613268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.613550 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.613551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.613619 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.614020 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.614025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.615536 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260201 03:09:06.615650 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.612248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:40227 (local address 127.28.181.62:38983)
0201 03:09:06.612294 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.612295 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.612301 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:09:06.612315 (+ 14us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.612316 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.612321 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.612369 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.612662 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.613199 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.613619 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.613765 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.615596 (+ 1831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:40227: BlockingRecv error: recv got EOF from 127.28.181.3:40227 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.618729 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.613000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.28.181.62:38983)
0201 03:09:06.613062 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.613064 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.613182 (+ 118us) server_negotiation.cc:408] Connection header received
0201 03:09:06.613258 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.613259 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.613268 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.613332 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.614114 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.614526 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.615053 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.615211 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.615947 (+ 736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:06.615951 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:06.615951 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:06.615984 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:06.617496 (+ 1512us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:06.617850 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:06.617853 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:06.617853 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:06.617920 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:06.618107 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:06.618108 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:06.618108 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:06.618252 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:06.618359 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:06.618528 (+ 169us) server_negotiation.cc:300] Negotiation successful
0201 03:09:06.618641 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Feb 01 03:09:06 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:06.619745 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:06.615981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:44273)
0201 03:09:06.616075 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:06.616085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:06.616128 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:06.616258 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:06.616259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:06.616384 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:06.616519 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.616521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.617099 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.617100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:06.617558 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:06.617560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.617612 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.617977 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:06.617981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:06.619685 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260201 03:09:06.619788 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:06.616036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:44273 (local address 127.28.181.62:38983)
0201 03:09:06.616079 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:06.616081 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:06.616116 (+ 35us) server_negotiation.cc:408] Connection header received
0201 03:09:06.616193 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:06.616193 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:06.616201 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:06.616251 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:06.616588 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.617032 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:06.617620 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:06.617756 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:06.619742 (+ 1986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:44273: BlockingRecv error: recv got EOF from 127.28.181.3:44273 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260201 03:09:06.619935 6730 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:07.396761 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:07.392664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:58607)
0201 03:09:07.392758 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:07.392766 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:07.392803 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:07.393455 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:07.393455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:07.393654 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:07.393882 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.393884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.394450 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.394451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:07.394774 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.394775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.394824 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.395235 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:07.395239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:07.396677 (+ 1438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":53,"threads_started":1}
W20260201 03:09:07.396806 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:07.392697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:58607 (local address 127.28.181.62:38983)
0201 03:09:07.393338 (+ 641us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:07.393340 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:07.393349 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:09:07.393368 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:07.393369 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:07.393379 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:07.393437 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:07.393958 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.394392 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.394830 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.394936 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.396746 (+ 1810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:58607: BlockingRecv error: recv got EOF from 127.28.181.1:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":86,"threads_started":1}
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:07.520290 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:07.516517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:55295)
0201 03:09:07.516651 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:07.516661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:07.516737 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:07.516856 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:07.516857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:07.516997 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:07.517143 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.517144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.517699 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.517700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:07.518093 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.518095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.518195 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.518662 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:07.518665 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:07.520239 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
W20260201 03:09:07.520398 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:07.516589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:55295 (local address 127.28.181.62:38983)
0201 03:09:07.516648 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:07.516650 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:07.516722 (+ 72us) server_negotiation.cc:408] Connection header received
0201 03:09:07.516783 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:07.516784 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:07.516800 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:07.516853 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:07.517222 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.517629 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.518194 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.518392 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.520316 (+ 1924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:55295: BlockingRecv error: recv got EOF from 127.28.181.2:55295 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:07 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:07.623921 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:07.620304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:33053)
0201 03:09:07.620443 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:07.620454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:07.620528 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:07.620681 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:07.620682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:07.620826 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:07.620973 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.620975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.621527 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.621528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:07.621886 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:07.621887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.621964 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.622380 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:07.622383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:07.623830 (+ 1447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
W20260201 03:09:07.623946 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:07.620390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:33053 (local address 127.28.181.62:38983)
0201 03:09:07.620447 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:07.620449 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:07.620513 (+ 64us) server_negotiation.cc:408] Connection header received
0201 03:09:07.620590 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:07.620591 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:07.620601 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:07.620665 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:07.621050 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.621457 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:07.621973 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:07.622128 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:07.623893 (+ 1765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:33053: BlockingRecv error: recv got EOF from 127.28.181.3:33053 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:08.400508 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:08.397136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:55145)
0201 03:09:08.397247 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:08.397257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:08.397310 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:08.397427 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:08.397427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:08.397559 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:08.397701 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.397702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.398272 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.398273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:08.398623 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.398624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.398674 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.399070 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:08.399073 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:08.400463 (+ 1390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
W20260201 03:09:08.400609 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:08.397192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:55145 (local address 127.28.181.62:38983)
0201 03:09:08.397297 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:08.397299 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:08.397307 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:08.397352 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:08.397352 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:08.397360 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:08.397410 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:08.397754 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.398217 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.398684 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.398782 (+ 98us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.400542 (+ 1760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:55145: BlockingRecv error: recv got EOF from 127.28.181.1:55145 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":66,"threads_started":1}
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:08.524547 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:08.520816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:53375)
0201 03:09:08.520988 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:08.520999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:08.521076 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:08.521226 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:08.521227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:08.521379 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:08.521516 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.521518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.522149 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.522150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:08.522471 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.522473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.522545 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.523025 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:08.523028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:08.524473 (+ 1445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260201 03:09:08.524593 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:08.520901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:53375 (local address 127.28.181.62:38983)
0201 03:09:08.520983 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:08.520985 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:08.521059 (+ 74us) server_negotiation.cc:408] Connection header received
0201 03:09:08.521147 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:08.521148 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:08.521159 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:08.521221 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:08.521602 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.522079 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.522544 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.522703 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.524539 (+ 1836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:53375: BlockingRecv error: recv got EOF from 127.28.181.2:53375 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:08 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:08.627825 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:08.624489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:49525)
0201 03:09:08.624617 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:08.624628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:08.624686 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:08.624795 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:08.624796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:08.624921 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:08.625054 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.625055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.625549 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.625550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:08.625880 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:08.625881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.625927 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.626331 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:08.626335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:08.627780 (+ 1445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
W20260201 03:09:08.627955 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:08.624604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:49525 (local address 127.28.181.62:38983)
0201 03:09:08.624653 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:08.624655 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:08.624673 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:09:08.624730 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:08.624731 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:08.624738 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:08.624779 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:08.625106 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.625496 (+ 390us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:08.625937 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:08.626038 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:08.627880 (+ 1842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:49525: BlockingRecv error: recv got EOF from 127.28.181.3:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:09.419795 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:09.401095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:32811)
0201 03:09:09.401296 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:09.401304 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:09.401371 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:09.401510 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:09.401511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:09.401661 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:09.401864 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.401866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.402517 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.402518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:09.402879 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.402880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.402987 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.418188 (+ 15201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:09.418192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:09.419748 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
W20260201 03:09:09.419916 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:09.401179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:32811 (local address 127.28.181.62:38983)
0201 03:09:09.401294 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:09.401296 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:09.401357 (+ 61us) server_negotiation.cc:408] Connection header received
0201 03:09:09.401423 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:09.401423 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:09.401435 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:09.401504 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:09.401946 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.402436 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.402985 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.403175 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.419837 (+ 16662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:32811: BlockingRecv error: recv got EOF from 127.28.181.1:32811 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:09.528719 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:09.525021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:43485)
0201 03:09:09.525175 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:09.525185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:09.525266 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:09.525392 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:09.525392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:09.525543 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:09.525676 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.525678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.526275 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.526276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:09.526591 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.526593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.526664 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.527130 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:09.527133 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:09.528642 (+ 1509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
W20260201 03:09:09.528769 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:09.525111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:43485 (local address 127.28.181.62:38983)
0201 03:09:09.525172 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:09.525174 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:09.525249 (+ 75us) server_negotiation.cc:408] Connection header received
0201 03:09:09.525311 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:09.525311 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:09.525323 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:09.525385 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:09.525749 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.526197 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.526662 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.526820 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.528710 (+ 1890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:43485: BlockingRecv error: recv got EOF from 127.28.181.2:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:09 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:09.631434 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:09.628147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:41883)
0201 03:09:09.628259 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:09.628268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:09.628313 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:09.628439 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:09.628440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:09.628568 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:09.628702 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.628704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.629246 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.629247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:09.629563 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:09.629564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.629612 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.630004 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:09.630007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:09.631390 (+ 1383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
W20260201 03:09:09.631539 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:09.628216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:41883 (local address 127.28.181.62:38983)
0201 03:09:09.628267 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:09.628269 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:09.628303 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:09:09.628355 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:09.628355 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:09.628363 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:09.628410 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:09.628759 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.629180 (+ 421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:09.629621 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:09.629723 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:09.631470 (+ 1747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:41883: BlockingRecv error: recv got EOF from 127.28.181.3:41883 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:10.423940 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:10.420348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:47241)
0201 03:09:10.420467 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:10.420477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:10.420518 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:10.420628 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:10.420628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:10.420761 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:10.420900 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.420901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.421449 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.421449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:10.421957 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.421959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.422010 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.422437 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:10.422440 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:10.423889 (+ 1449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
W20260201 03:09:10.424075 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:10.420419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:47241 (local address 127.28.181.62:38983)
0201 03:09:10.420523 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:10.420524 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:10.420533 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:09:10.420553 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:10.420553 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:10.420561 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:10.420615 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:10.420963 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.421391 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.422024 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.422156 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.423987 (+ 1831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:47241: BlockingRecv error: recv got EOF from 127.28.181.1:47241 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:10.532956 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:10.529129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:46095)
0201 03:09:10.529315 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:10.529343 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:10.529407 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:10.529563 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:10.529564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:10.529758 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:10.529982 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.529984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.530603 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.530603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:10.530905 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.530906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.530956 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.531455 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:10.531459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:10.532911 (+ 1452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
W20260201 03:09:10.533068 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:10.529241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:46095 (local address 127.28.181.62:38983)
0201 03:09:10.529300 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:10.529302 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:10.529393 (+ 91us) server_negotiation.cc:408] Connection header received
0201 03:09:10.529467 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:10.529468 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:10.529497 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:10.529538 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:10.530069 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.530542 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.530977 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.531072 (+ 95us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.533001 (+ 1929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:46095: BlockingRecv error: recv got EOF from 127.28.181.2:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:10 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:10.635542 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:10.631939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:59887)
0201 03:09:10.632082 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:10.632090 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:10.632159 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:10.632308 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:10.632309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:10.632460 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:10.632601 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.632602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.633330 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.633331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:10.633646 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:10.633647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.633722 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.634114 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:10.634117 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:10.635497 (+ 1380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
W20260201 03:09:10.635649 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:10.631988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:59887 (local address 127.28.181.62:38983)
0201 03:09:10.632047 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:10.632049 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:10.632145 (+ 96us) server_negotiation.cc:408] Connection header received
0201 03:09:10.632235 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:10.632235 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:10.632250 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:10.632303 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:10.632674 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.633264 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:10.633720 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:10.633900 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:10.635578 (+ 1678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:59887: BlockingRecv error: recv got EOF from 127.28.181.3:59887 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:11.428689 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:11.424396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:57579)
0201 03:09:11.424517 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:11.424533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:11.424576 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:11.424712 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:11.424712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:11.424852 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:11.425020 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.425021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.426037 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.426038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:11.426352 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.426353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.426401 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.426874 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:11.426878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:11.428636 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
W20260201 03:09:11.428831 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:11.424461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:57579 (local address 127.28.181.62:38983)
0201 03:09:11.424568 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:11.424571 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:11.424582 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:09:11.424612 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:11.424613 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:11.424625 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:11.424703 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:11.425099 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.425971 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.426421 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.426608 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.428745 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:57579: BlockingRecv error: recv got EOF from 127.28.181.1:57579 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:11.537213 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:11.533516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:42609)
0201 03:09:11.533663 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:11.533672 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:11.533723 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:11.533839 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:11.533839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:11.533964 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:11.534105 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.534106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.534617 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.534618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:11.534948 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.534949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.535007 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.535443 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:11.535447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:11.537145 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260201 03:09:11.537272 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:11.533617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:42609 (local address 127.28.181.62:38983)
0201 03:09:11.533672 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:11.533674 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:11.533723 (+ 49us) server_negotiation.cc:408] Connection header received
0201 03:09:11.533759 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:11.533759 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:11.533781 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:11.533820 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:11.534164 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.534563 (+ 399us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.535020 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.535121 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.537219 (+ 2098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:42609: BlockingRecv error: recv got EOF from 127.28.181.2:42609 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:11 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:11.639689 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:11.635922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:44787)
0201 03:09:11.636077 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:11.636087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:11.636164 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:11.636297 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:11.636297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:11.636453 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:11.636591 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.636593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.637185 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.637186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:11.637510 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:11.637511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.637584 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.638089 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:11.638092 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:11.639612 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
W20260201 03:09:11.639750 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:11.635991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:44787 (local address 127.28.181.62:38983)
0201 03:09:11.636070 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:11.636072 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:11.636148 (+ 76us) server_negotiation.cc:408] Connection header received
0201 03:09:11.636235 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:11.636235 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:11.636246 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:11.636292 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:11.636666 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.637105 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:11.637582 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:11.637745 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:11.639685 (+ 1940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:44787: BlockingRecv error: recv got EOF from 127.28.181.3:44787 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:12.432715 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:12.429236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:43631)
0201 03:09:12.429347 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:12.429355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:12.429397 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:12.429517 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:12.429517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:12.429658 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:12.429851 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.429853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.430442 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.430443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:12.430763 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.430764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.430810 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.431207 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:12.431211 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:12.432659 (+ 1448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
W20260201 03:09:12.432806 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:12.429275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:43631 (local address 127.28.181.62:38983)
0201 03:09:12.429392 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:12.429393 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:12.429401 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:12.429444 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:12.429444 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:12.429453 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:12.429505 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:12.429919 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.430385 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.430817 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.430911 (+ 94us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.432741 (+ 1830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:43631: BlockingRecv error: recv got EOF from 127.28.181.1:43631 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:12.541224 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:12.537767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:55919)
0201 03:09:12.537903 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:12.537916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:12.537968 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:12.538086 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:12.538087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:12.538217 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:12.538355 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.538356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.538902 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.538903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:12.539240 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.539241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.539290 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.539748 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:12.539752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:12.541171 (+ 1419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260201 03:09:12.541332 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:12.537867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:55919 (local address 127.28.181.62:38983)
0201 03:09:12.537920 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:12.537922 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:12.537956 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:09:12.538015 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:12.538015 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:12.538024 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:12.538064 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:12.538410 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.538842 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.539298 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.539391 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.541247 (+ 1856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:55919: BlockingRecv error: recv got EOF from 127.28.181.2:55919 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:12 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:12.643296 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:12.640078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:54293)
0201 03:09:12.640174 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:12.640181 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:12.640225 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:12.640325 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:12.640325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:12.640452 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:12.640586 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.640588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.641093 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.641093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:12.641400 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:12.641401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.641448 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.641847 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:12.641851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:12.643247 (+ 1396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
W20260201 03:09:12.643396 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:12.640117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:54293 (local address 127.28.181.62:38983)
0201 03:09:12.640167 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:12.640168 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:12.640225 (+ 57us) server_negotiation.cc:408] Connection header received
0201 03:09:12.640259 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:12.640260 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:12.640267 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:12.640310 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:12.640635 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.641037 (+ 402us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:12.641453 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:12.641543 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:12.643328 (+ 1785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:54293: BlockingRecv error: recv got EOF from 127.28.181.3:54293 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:13.436766 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:13.433154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:37039)
0201 03:09:13.433272 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:13.433282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:13.433331 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:13.433454 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:13.433455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:13.433591 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:13.433732 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.433733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.434300 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.434301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:13.434664 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.434665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.434719 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.435154 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:13.435158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:13.436715 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
W20260201 03:09:13.436883 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:13.433208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:37039 (local address 127.28.181.62:38983)
0201 03:09:13.433308 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:13.433310 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:13.433321 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:09:13.433386 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:13.433386 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:13.433394 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:13.433436 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:13.433808 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.434242 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.434729 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.434836 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.436800 (+ 1964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:37039: BlockingRecv error: recv got EOF from 127.28.181.1:37039 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:13.545166 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:13.541610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:42881)
0201 03:09:13.541828 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:13.541840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:13.541903 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:13.542015 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:13.542015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:13.542144 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:13.542278 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.542279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.542787 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.542787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:13.543095 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.543096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.543141 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.543634 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:13.543638 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:13.545119 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
W20260201 03:09:13.545253 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:13.541816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:42881 (local address 127.28.181.62:38983)
0201 03:09:13.541864 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:13.541865 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:13.541888 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:09:13.541939 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:13.541940 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:13.541949 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:13.541995 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:13.542332 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.542730 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.543157 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.543301 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.545197 (+ 1896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:42881: BlockingRecv error: recv got EOF from 127.28.181.2:42881 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:13 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:13.647732 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:13.643807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:38241)
0201 03:09:13.643931 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:13.643940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:13.643982 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:13.644155 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:13.644156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:13.644313 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:13.644446 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.644448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.645011 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.645012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:13.645387 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:13.645388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.645470 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.645938 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:13.645942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:13.647649 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
W20260201 03:09:13.647773 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:13.643991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:38241 (local address 127.28.181.62:38983)
0201 03:09:13.644050 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:13.644051 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:13.644064 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:09:13.644079 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:13.644080 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:13.644087 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:13.644136 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:13.644521 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.644931 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:13.645468 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:13.645664 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:13.647717 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:38241: BlockingRecv error: recv got EOF from 127.28.181.3:38241 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:14.446339 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:14.437320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:37019)
0201 03:09:14.437456 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:14.437466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:14.437513 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:14.437808 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:14.437809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:14.437966 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:14.438132 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.438133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.438737 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.438738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:14.439056 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.439057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.439103 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.444624 (+ 5521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:14.444628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:14.446257 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
W20260201 03:09:14.446389 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:14.437523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:37019 (local address 127.28.181.62:38983)
0201 03:09:14.437655 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:14.437657 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:14.437665 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:14.437684 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:14.437685 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:14.437695 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:14.437755 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:14.438197 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.438679 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.439111 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.439202 (+ 91us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.446331 (+ 7129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:37019: BlockingRecv error: recv got EOF from 127.28.181.1:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:14.550046 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:14.545825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:57425)
0201 03:09:14.545971 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:14.545983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:14.546075 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:14.546249 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:14.546250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:14.546416 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:14.546564 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.546566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.547368 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.547369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:14.547730 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.547731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.547797 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.548200 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:14.548204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:14.549948 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
W20260201 03:09:14.550081 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:14.545856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:57425 (local address 127.28.181.62:38983)
0201 03:09:14.545957 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:14.545960 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:14.546057 (+ 97us) server_negotiation.cc:408] Connection header received
0201 03:09:14.546153 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:14.546154 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:14.546168 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:14.546227 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:14.546645 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.547295 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.547804 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.547976 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.550022 (+ 2046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:57425: BlockingRecv error: recv got EOF from 127.28.181.2:57425 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:14 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:14.651924 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:14.648187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:35713)
0201 03:09:14.648324 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:14.648334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:14.648388 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:14.648504 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:14.648504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:14.648647 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:14.648795 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.648797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.649390 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.649391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:14.649710 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:14.649711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.649758 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.650293 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:14.650297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:14.651875 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260201 03:09:14.652017 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:14.648262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:35713 (local address 127.28.181.62:38983)
0201 03:09:14.648335 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:14.648337 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:14.648375 (+ 38us) server_negotiation.cc:408] Connection header received
0201 03:09:14.648440 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:14.648440 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:14.648448 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:14.648487 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:14.648860 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.649329 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:14.649800 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:14.649906 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:14.651957 (+ 2051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:35713: BlockingRecv error: recv got EOF from 127.28.181.3:35713 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:15.450392 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:15.446757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:53247)
0201 03:09:15.446886 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:15.446895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:15.446943 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:15.447066 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:15.447067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:15.447246 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:15.447408 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.447409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.448017 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.448017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:15.448347 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.448348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.448396 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.448815 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:15.448819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:15.450342 (+ 1523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260201 03:09:15.450492 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:15.446832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:53247 (local address 127.28.181.62:38983)
0201 03:09:15.446944 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:15.446946 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:15.446979 (+ 33us) server_negotiation.cc:408] Connection header received
0201 03:09:15.446997 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:15.446998 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:15.447007 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:15.447051 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:15.447472 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.447960 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.448405 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.448562 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.450422 (+ 1860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:53247: BlockingRecv error: recv got EOF from 127.28.181.1:53247 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:15.554589 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:15.550598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:34007)
0201 03:09:15.550792 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:15.550806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:15.550890 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:15.551054 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:15.551055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:15.551233 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:15.551385 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.551387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.552063 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.552064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:15.552417 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.552418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.552484 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.552942 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:15.552946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:15.554541 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
W20260201 03:09:15.554682 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:15.550675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:34007 (local address 127.28.181.62:38983)
0201 03:09:15.550760 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:15.550763 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:15.550871 (+ 108us) server_negotiation.cc:408] Connection header received
0201 03:09:15.550961 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:15.550962 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:15.550975 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:15.551047 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:15.551467 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.551950 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.552495 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.552672 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.554618 (+ 1946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:34007: BlockingRecv error: recv got EOF from 127.28.181.2:34007 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:15 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:15.655813 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:15.652473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:53295)
0201 03:09:15.652583 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:15.652598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:15.652648 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:15.652747 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:15.652747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:15.652879 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:15.653017 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.653018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.653572 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.653573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:15.653906 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:15.653907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.653954 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.654358 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:15.654361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:15.655766 (+ 1405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
W20260201 03:09:15.655898 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:15.652525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:53295 (local address 127.28.181.62:38983)
0201 03:09:15.652584 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:15.652586 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:15.652650 (+ 64us) server_negotiation.cc:408] Connection header received
0201 03:09:15.652681 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:15.652682 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:15.652689 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:15.652734 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:15.653073 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.653512 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:15.653964 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:15.654064 (+ 100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:15.655843 (+ 1779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:53295: BlockingRecv error: recv got EOF from 127.28.181.3:53295 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:16.454739 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:16.450988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:49053)
0201 03:09:16.451126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:16.451135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:16.451214 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:16.451462 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:16.451463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:16.451666 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:16.451809 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.451811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.452388 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.452388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:16.452709 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.452710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.452758 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.453221 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:16.453227 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:16.454692 (+ 1465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260201 03:09:16.454854 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:16.451028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49053 (local address 127.28.181.62:38983)
0201 03:09:16.451166 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:16.451167 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:16.451200 (+ 33us) server_negotiation.cc:408] Connection header received
0201 03:09:16.451357 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:16.451358 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:16.451368 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:16.451441 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:16.451867 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.452326 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.452766 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.452873 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.454775 (+ 1902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:49053: BlockingRecv error: recv got EOF from 127.28.181.1:49053 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:16.558787 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:16.554974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:60637)
0201 03:09:16.555117 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:16.555127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:16.555204 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:16.555330 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:16.555331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:16.555478 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:16.555611 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.555613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.556180 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.556181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:16.556510 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.556511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.556584 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.557063 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:16.557066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:16.558710 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
W20260201 03:09:16.558831 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:16.555047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:60637 (local address 127.28.181.62:38983)
0201 03:09:16.555113 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:16.555115 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:16.555191 (+ 76us) server_negotiation.cc:408] Connection header received
0201 03:09:16.555261 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:16.555262 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:16.555273 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:16.555321 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:16.555687 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.556107 (+ 420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.556583 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.556750 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.558776 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:60637: BlockingRecv error: recv got EOF from 127.28.181.2:60637 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:16 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:16.660049 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:16.656326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:42183)
0201 03:09:16.656461 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:16.656471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:16.656519 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:16.656663 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:16.656663 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:16.656808 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:16.656939 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.656940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.657515 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.657517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:16.657851 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:16.657852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.657926 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.658433 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:16.658437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:16.659967 (+ 1530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":45,"threads_started":1}
W20260201 03:09:16.660100 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:16.656409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:42183 (local address 127.28.181.62:38983)
0201 03:09:16.656489 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:16.656491 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:16.656503 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:09:16.656587 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:16.656588 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:16.656602 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:16.656657 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:16.657026 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.657436 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:16.657924 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:16.658100 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:16.660043 (+ 1943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:42183: BlockingRecv error: recv got EOF from 127.28.181.3:42183 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:17.458503 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:17.455155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:56523)
0201 03:09:17.455287 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:17.455296 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:17.455343 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:17.455473 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:17.455474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:17.455606 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:17.455746 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.455747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.456290 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.456291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:17.456598 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.456599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.456644 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.457028 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:17.457031 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:17.458456 (+ 1425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
W20260201 03:09:17.458617 6775 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:17.455224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:56523 (local address 127.28.181.62:38983)
0201 03:09:17.455332 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:17.455333 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:17.455341 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:17.455393 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:17.455393 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:17.455402 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:17.455445 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:17.455802 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.456233 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.456653 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.456746 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.458539 (+ 1793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:56523: BlockingRecv error: recv got EOF from 127.28.181.1:56523 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:17.562778 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:17.559288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:32833)
0201 03:09:17.559420 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:17.559428 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:17.559478 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:17.559591 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:17.559592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:17.559722 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:17.559861 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.559862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.560407 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.560407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:17.560777 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.560779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.560828 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.561245 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:17.561249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:17.562731 (+ 1482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
W20260201 03:09:17.562891 6775 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:17.559343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:32833 (local address 127.28.181.62:38983)
0201 03:09:17.559396 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:17.559397 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:17.559475 (+ 78us) server_negotiation.cc:408] Connection header received
0201 03:09:17.559520 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:17.559521 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:17.559530 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:17.559575 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:17.559913 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.560349 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.560831 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.560921 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.562822 (+ 1901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:32833: BlockingRecv error: recv got EOF from 127.28.181.2:32833 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:17 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:17.663983 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:17.660453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:52043)
0201 03:09:17.660574 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:17.660583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:17.660622 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:17.660746 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:17.660747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:17.660873 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:17.661003 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.661004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.661502 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.661502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:17.661885 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:17.661887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.661937 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.662373 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:17.662377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:17.663930 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
W20260201 03:09:17.664042 6775 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:17.660597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:52043 (local address 127.28.181.62:38983)
0201 03:09:17.660653 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:17.660655 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:17.660662 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:17.660678 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:17.660679 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:17.660686 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:17.660732 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:17.661056 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.661448 (+ 392us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:17.661940 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:17.662068 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:17.664002 (+ 1934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:52043: BlockingRecv error: recv got EOF from 127.28.181.3:52043 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:18.462455 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:18.458921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:39761)
0201 03:09:18.459063 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:18.459072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:18.459129 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:18.459255 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:18.459256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:18.459402 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:18.459545 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.459546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.460079 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.460079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:18.460433 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.460434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.460479 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.460880 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:18.460883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:18.462404 (+ 1521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260201 03:09:18.462556 6779 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:18.458999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:39761 (local address 127.28.181.62:38983)
0201 03:09:18.459084 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:18.459089 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:18.459112 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:09:18.459182 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:18.459183 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:18.459191 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:18.459240 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:18.459596 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.460021 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.460485 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.460575 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.462486 (+ 1911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:39761: BlockingRecv error: recv got EOF from 127.28.181.1:39761 (error 108)
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":47,"threads_started":1}
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:18.566570 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:18.563251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:40419)
0201 03:09:18.563377 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:18.563385 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:18.563429 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:18.563534 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:18.563535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:18.563672 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:18.563811 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.563813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.564335 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.564335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:18.564640 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.564641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.564688 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.565078 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:18.565082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:18.566520 (+ 1438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
W20260201 03:09:18.566691 6779 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:18.563370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:40419 (local address 127.28.181.62:38983)
0201 03:09:18.563438 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:18.563440 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:18.563447 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:18.563465 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:18.563465 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:18.563476 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:18.563523 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:18.563866 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.564278 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.564693 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.564781 (+ 88us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.566613 (+ 1832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:40419: BlockingRecv error: recv got EOF from 127.28.181.2:40419 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:18 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:18.667858 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:18.664490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:38315)
0201 03:09:18.664647 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:18.664654 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:18.664705 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:18.664805 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:18.664806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:18.664942 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:18.665082 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.665083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.665602 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.665602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:18.665952 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:18.665954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.666004 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.666443 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:18.666446 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:18.667807 (+ 1361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260201 03:09:18.667943 6779 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:18.664541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:38315 (local address 127.28.181.62:38983)
0201 03:09:18.664599 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:18.664601 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:18.664706 (+ 105us) server_negotiation.cc:408] Connection header received
0201 03:09:18.664738 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:18.664739 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:18.664747 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:18.664788 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:18.665140 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.665544 (+ 404us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:18.666012 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:18.666111 (+ 99us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:18.667884 (+ 1773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:38315: BlockingRecv error: recv got EOF from 127.28.181.3:38315 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:19.470638 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:19.462858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:54905)
0201 03:09:19.462989 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:19.462997 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:19.463047 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:19.463163 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:19.463164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:19.463297 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:19.463445 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.463447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.464280 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.464281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:19.464606 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.464607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.464658 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.468891 (+ 4233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:19.468895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:19.470555 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260201 03:09:19.470672 6783 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:19.462896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:54905 (local address 127.28.181.62:38983)
0201 03:09:19.463001 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:19.463003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:19.463054 (+ 51us) server_negotiation.cc:408] Connection header received
0201 03:09:19.463086 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:19.463086 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:19.463095 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:19.463148 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:19.463540 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.464216 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.464665 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.464758 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.470618 (+ 5860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:54905: BlockingRecv error: recv got EOF from 127.28.181.1:54905 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:19.570595 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:19.567065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:36077)
0201 03:09:19.567180 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:19.567189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:19.567229 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:19.567349 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:19.567350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:19.567479 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:19.567617 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.567618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.568139 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.568139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:19.568469 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.568470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.568517 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.568932 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:19.568936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:19.570498 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
W20260201 03:09:19.570657 6783 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:19.567146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:36077 (local address 127.28.181.62:38983)
0201 03:09:19.567206 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:19.567208 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:19.567219 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:09:19.567279 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:19.567279 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:19.567290 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:19.567335 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:19.567671 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.568082 (+ 411us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.568526 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.568619 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.570596 (+ 1977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:36077: BlockingRecv error: recv got EOF from 127.28.181.2:36077 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:19 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:19.671736 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:19.668250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:46577)
0201 03:09:19.668365 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:19.668374 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:19.668425 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:19.668528 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:19.668529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:19.668667 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:19.668803 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.668805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.669345 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.669346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:19.669667 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:19.669668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.669717 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.670221 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:19.670225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:19.671686 (+ 1461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
W20260201 03:09:19.671859 6783 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:19.668304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:46577 (local address 127.28.181.62:38983)
0201 03:09:19.668371 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:19.668373 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:19.668419 (+ 46us) server_negotiation.cc:408] Connection header received
0201 03:09:19.668462 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:19.668462 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:19.668470 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:19.668511 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:19.668857 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.669267 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:19.669725 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:19.669839 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:19.671790 (+ 1951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:46577: BlockingRecv error: recv got EOF from 127.28.181.3:46577 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:20.474704 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:20.471124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:37907)
0201 03:09:20.471261 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:20.471270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:20.471311 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:20.471427 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:20.471427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:20.471612 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:20.471751 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.471753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.472419 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.472420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:20.472737 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.472738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.472787 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.473187 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:20.473191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:20.474656 (+ 1465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260201 03:09:20.474819 6787 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:20.471206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:37907 (local address 127.28.181.62:38983)
0201 03:09:20.471310 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:20.471318 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:20.471326 (+ 8us) server_negotiation.cc:408] Connection header received
0201 03:09:20.471347 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:20.471347 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:20.471356 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:20.471411 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:20.471848 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.472360 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.472796 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.472891 (+ 95us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.474741 (+ 1850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:37907: BlockingRecv error: recv got EOF from 127.28.181.1:37907 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:20.574560 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:20.570957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:56719)
0201 03:09:20.571096 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:20.571106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:20.571181 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:20.571329 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:20.571329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:20.571485 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:20.571620 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.571621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.572199 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.572200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:20.572527 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.572529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.572600 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.573011 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:20.573014 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:20.574475 (+ 1461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260201 03:09:20.574601 6787 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:20.571024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:56719 (local address 127.28.181.62:38983)
0201 03:09:20.571090 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:20.571092 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:20.571163 (+ 71us) server_negotiation.cc:408] Connection header received
0201 03:09:20.571241 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:20.571242 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:20.571255 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:20.571317 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:20.571698 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.572126 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.572598 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.572761 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.574545 (+ 1784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:56719: BlockingRecv error: recv got EOF from 127.28.181.2:56719 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:20 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.3@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:20.675822 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:20.672250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.3:49649)
0201 03:09:20.672402 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:20.672411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:20.672466 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:20.672565 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:20.672566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:20.672706 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:20.672842 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.672843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.673363 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.673364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:20.673745 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:20.673748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.673837 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.674347 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:20.674351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:20.675772 (+ 1421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260201 03:09:20.675940 6787 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:20.672299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:49649 (local address 127.28.181.62:38983)
0201 03:09:20.672364 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:20.672366 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:20.672465 (+ 99us) server_negotiation.cc:408] Connection header received
0201 03:09:20.672503 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:20.672504 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:20.672512 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:20.672554 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:20.672896 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.673302 (+ 406us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:20.673860 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:20.674039 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:20.675866 (+ 1827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.3:49649: BlockingRecv error: recv got EOF from 127.28.181.3:49649 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Feb 01 03:09:21 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:21 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.1@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:21.478708 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:21.475154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.1:51043)
0201 03:09:21.475283 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:21.475291 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:21.475339 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:21.475458 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:21.475458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:21.475594 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:21.475734 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.475735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.476264 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.476265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:21.476570 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.476571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.476616 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.477051 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:21.477054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:21.478623 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260201 03:09:21.478791 6791 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:21.475238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:51043 (local address 127.28.181.62:38983)
0201 03:09:21.475359 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:21.475360 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:21.475367 (+ 7us) server_negotiation.cc:408] Connection header received
0201 03:09:21.475385 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:21.475385 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:21.475394 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:21.475444 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:21.475788 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.476209 (+ 421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.476623 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.476717 (+ 94us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.478713 (+ 1996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.1:51043: BlockingRecv error: recv got EOF from 127.28.181.1:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
Feb 01 03:09:21 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:09:21 dist-test-slave-ck1n krb5kdc[5723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.181.2@KRBTEST.COM for oryx/127.28.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:09:21.578977 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:21.575001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38983 (local address 127.28.181.2:43941)
0201 03:09:21.575138 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:21.575148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:21.575212 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:21.575332 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:21.575333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:21.575502 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:21.575635 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.575637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.576261 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.576262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:21.576576 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.576577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.576648 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.577184 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:21.577188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:21.578929 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.181.62:38983: Server oryx/127.28.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
W20260201 03:09:21.579097 6791 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:21.575060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:43941 (local address 127.28.181.62:38983)
0201 03:09:21.575124 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:21.575126 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:21.575202 (+ 76us) server_negotiation.cc:408] Connection header received
0201 03:09:21.575267 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:21.575267 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:21.575278 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:21.575326 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:21.575712 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.576186 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.576644 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.576823 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.579030 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.181.2:43941: BlockingRecv error: recv got EOF from 127.28.181.2:43941 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:09:21.617062 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6315
I20260201 03:09:21.621171 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6468
I20260201 03:09:21.624850 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6600
I20260201 03:09:21.628687 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6245
2026-02-01T03:09:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17899 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-02-01T03:09:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:21Z Disabled control of system clock
I20260201 03:09:21.652981 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:34907
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:34907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260201 03:09:21.719354 6796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:21.719488 6796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:21.719504 6796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:21.720703 6796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:21.720734 6796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:21.720744 6796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:21.720755 6796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:21.720765 6796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:21.722003 6796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:34907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:34907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:21.722141 6796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:21.722281 6796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:21.724440 6801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:21.724457 6802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:21.724489 6796 server_base.cc:1060] running on GCE node
W20260201 03:09:21.724447 6804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:21.724787 6796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:21.724975 6796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:21.726094 6796 hybrid_clock.cc:648] HybridClock initialized: now 1769915361726083 us; error 25 us; skew 500 ppm
I20260201 03:09:21.726976 6796 webserver.cc:492] Webserver started at http://127.28.181.62:45831/ using document root <none> and password file <none>
I20260201 03:09:21.727123 6796 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:21.727152 6796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:21.727216 6796 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:21.727917 6796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "ada35f8a91b54c3fa9bde5391ff83f57"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "f24a62814e3f83d56881c0390b6f85ed"
server_key_iv: "b6a4e776c7bd9b0404fc68dd96d6534a"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.728148 6796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "ada35f8a91b54c3fa9bde5391ff83f57"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "f24a62814e3f83d56881c0390b6f85ed"
server_key_iv: "b6a4e776c7bd9b0404fc68dd96d6534a"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.729041 6796 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:21.729588 6810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.729735 6796 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:21.729843 6796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "ada35f8a91b54c3fa9bde5391ff83f57"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "f24a62814e3f83d56881c0390b6f85ed"
server_key_iv: "b6a4e776c7bd9b0404fc68dd96d6534a"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.729892 6796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:21.750934 6796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:21.751097 6796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:21.751166 6796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:21.754099 6796 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:34907
I20260201 03:09:21.754125 6862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:34907 every 8 connection(s)
I20260201 03:09:21.754372 6796 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:21.754901 6863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:21.755898 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6796
I20260201 03:09:21.755968 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:09:21.756119 29396 external_mini_cluster.cc:1468] Setting key d86048ab6415a9ff42abea132145afc7
I20260201 03:09:21.757084 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57: Bootstrap starting.
I20260201 03:09:21.757577 6863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:21.757869 6863 log.cc:826] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:21.758467 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57: No bootstrap required, opened a new log
I20260201 03:09:21.759501 6863 raft_consensus.cc:359] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } }
I20260201 03:09:21.759630 6863 raft_consensus.cc:385] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:21.759670 6863 raft_consensus.cc:740] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada35f8a91b54c3fa9bde5391ff83f57, State: Initialized, Role: FOLLOWER
I20260201 03:09:21.759768 6863 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } }
I20260201 03:09:21.759843 6863 raft_consensus.cc:399] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:21.759892 6863 raft_consensus.cc:493] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:21.759935 6863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:21.760090 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:21.756570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.28.181.62:34907)
0201 03:09:21.756756 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:21.756762 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:21.756783 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:09:21.757039 (+ 256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:21.757044 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:21.757083 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:21.757196 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:21.757543 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.758410 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.758950 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.759160 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.759494 (+ 334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:21.759501 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:21.759501 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:21.759688 (+ 187us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:21.759715 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:21.759722 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:21.759777 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:21.759841 (+ 64us) server_negotiation.cc:300] Negotiation successful
0201 03:09:21.759966 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260201 03:09:21.760519 6863 raft_consensus.cc:515] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } }
I20260201 03:09:21.760648 6863 leader_election.cc:304] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada35f8a91b54c3fa9bde5391ff83f57; no voters:
I20260201 03:09:21.760776 6863 leader_election.cc:290] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:09:21.760816 6868 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:21.760928 6868 raft_consensus.cc:697] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [term 1 LEADER]: Becoming Leader. State: Replica: ada35f8a91b54c3fa9bde5391ff83f57, State: Running, Role: LEADER
I20260201 03:09:21.760962 6863 sys_catalog.cc:565] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:21.761014 6868 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } }
I20260201 03:09:21.761337 6870 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada35f8a91b54c3fa9bde5391ff83f57. Latest consensus state: current_term: 1 leader_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } } }
I20260201 03:09:21.761363 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada35f8a91b54c3fa9bde5391ff83f57" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34907 } } }
I20260201 03:09:21.761499 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:21.761840 6877 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:21.761862 6870 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:21.762279 6877 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:09:21.763398 6877 catalog_manager.cc:1357] Generated new cluster ID: ad36f75fb433460a9ba1c2956bf1d9f0
I20260201 03:09:21.763461 6877 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:09:21.772984 6877 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:09:21.773381 6877 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:21.787504 6877 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ada35f8a91b54c3fa9bde5391ff83f57: Generated new TSK 0
I20260201 03:09:21.787655 6877 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:09:21.789172 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--builtin_ntp_servers=127.28.181.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260201 03:09:21.855407 6887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:21.855535 6887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:21.855552 6887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:21.856714 6887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:21.856750 6887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:21.856770 6887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:21.858122 6887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:21.858304 6887 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:21.858506 6887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:21.860718 6893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:21.860826 6887 server_base.cc:1060] running on GCE node
W20260201 03:09:21.860710 6892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:21.860694 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:21.861053 6887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:21.861244 6887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:21.862373 6887 hybrid_clock.cc:648] HybridClock initialized: now 1769915361862365 us; error 23 us; skew 500 ppm
I20260201 03:09:21.863332 6887 webserver.cc:492] Webserver started at http://127.28.181.1:34321/ using document root <none> and password file <none>
I20260201 03:09:21.863510 6887 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:21.863543 6887 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:21.863641 6887 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:21.864396 6887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "709236ab2c864bae94efdc7b21fba1f9"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "3b6e11186496a6d3db76db2f87df2173"
server_key_iv: "0ffed1bb6473edb04f0900fff7a8fa07"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.864657 6887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "709236ab2c864bae94efdc7b21fba1f9"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "3b6e11186496a6d3db76db2f87df2173"
server_key_iv: "0ffed1bb6473edb04f0900fff7a8fa07"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.865630 6887 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:21.866324 6901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.866495 6887 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.866580 6887 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "709236ab2c864bae94efdc7b21fba1f9"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "3b6e11186496a6d3db76db2f87df2173"
server_key_iv: "0ffed1bb6473edb04f0900fff7a8fa07"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.866644 6887 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:21.892612 6887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:21.892807 6887 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:21.892915 6887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:21.893083 6887 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:21.893343 6887 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:21.893388 6887 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.893422 6887 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:21.893460 6887 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.898304 6887 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:34295
I20260201 03:09:21.898342 7014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:34295 every 8 connection(s)
I20260201 03:09:21.898577 6887 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:09:21.901762 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:21.899143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:40653 (local address 127.28.181.62:34907)
0201 03:09:21.899199 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:21.899201 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:21.899394 (+ 193us) server_negotiation.cc:408] Connection header received
0201 03:09:21.899788 (+ 394us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:21.899789 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:21.899798 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:21.899851 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:21.900220 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.900634 (+ 414us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.901081 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.901171 (+ 90us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.901561 (+ 390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:21.901565 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:21.901565 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:21.901569 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:21.901580 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:21.901582 (+ 2us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:21.901620 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:21.901688 (+ 68us) server_negotiation.cc:300] Negotiation successful
0201 03:09:21.901712 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260201 03:09:21.901896 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:21.899095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34907 (local address 127.28.181.1:40653)
0201 03:09:21.899309 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:21.899329 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:21.899723 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:21.899899 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:21.899901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:21.899908 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:21.900158 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.900160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.900708 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:21.900709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:21.901019 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:21.901020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:21.901100 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:21.901471 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:21.901472 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:21.901486 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:21.901499 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:21.901629 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:21.901632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:21.901634 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:21.901678 (+ 44us) client_negotiation.cc:240] Negotiation successful
0201 03:09:21.901724 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260201 03:09:21.902334 7015 heartbeater.cc:344] Connected to a master server at 127.28.181.62:34907
I20260201 03:09:21.902403 7015 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:21.902529 7015 heartbeater.cc:507] Master 127.28.181.62:34907 requested a full tablet report, sending...
I20260201 03:09:21.902770 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 6887
I20260201 03:09:21.902873 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:09:21.902927 6827 ts_manager.cc:194] Registered new tserver with Master: 709236ab2c864bae94efdc7b21fba1f9 (127.28.181.1:34295)
I20260201 03:09:21.903026 29396 external_mini_cluster.cc:1468] Setting key 11443b324ebc8cf9f15cf105adf50b59
I20260201 03:09:21.903693 6827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.181.1:40653
I20260201 03:09:21.904011 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--builtin_ntp_servers=127.28.181.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260201 03:09:21.969297 7018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:21.969535 7018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:21.969558 7018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:21.970805 7018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:21.970841 7018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:21.970885 7018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:21.972231 7018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:21.972369 7018 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:21.972566 7018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:21.974442 7024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:21.974465 7026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:21.974467 7023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:21.974706 7018 server_base.cc:1060] running on GCE node
I20260201 03:09:21.974860 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:21.975039 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:21.976173 7018 hybrid_clock.cc:648] HybridClock initialized: now 1769915361976163 us; error 20 us; skew 500 ppm
I20260201 03:09:21.977079 7018 webserver.cc:492] Webserver started at http://127.28.181.2:38645/ using document root <none> and password file <none>
I20260201 03:09:21.977246 7018 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:21.977308 7018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:21.977423 7018 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:21.978225 7018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "6978b78a2c2c4f718712707c345a235a"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "c8603737c3e21b67a2ad494405719251"
server_key_iv: "e3ad13752e1d2b561ff73c83bfacc131"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.978493 7018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "6978b78a2c2c4f718712707c345a235a"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "c8603737c3e21b67a2ad494405719251"
server_key_iv: "e3ad13752e1d2b561ff73c83bfacc131"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.979450 7018 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:21.980051 7032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.980199 7018 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:21.980281 7018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "6978b78a2c2c4f718712707c345a235a"
format_stamp: "Formatted at 2026-02-01 03:09:21 on dist-test-slave-ck1n"
server_key: "c8603737c3e21b67a2ad494405719251"
server_key_iv: "e3ad13752e1d2b561ff73c83bfacc131"
server_key_version: "encryptionkey@0"
I20260201 03:09:21.980348 7018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:21.996004 7018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:21.996176 7018 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:21.996285 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:21.996433 7018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:21.996668 7018 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:21.996712 7018 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:21.996742 7018 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:21.996764 7018 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.001412 7018 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:35507
I20260201 03:09:22.001470 7145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:35507 every 8 connection(s)
I20260201 03:09:22.001713 7018 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:09:22.005110 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.002350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:59007 (local address 127.28.181.62:34907)
0201 03:09:22.002447 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.002450 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.002599 (+ 149us) server_negotiation.cc:408] Connection header received
0201 03:09:22.002986 (+ 387us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.002987 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.002999 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.003054 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.003430 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.003853 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:22.004305 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.004402 (+ 97us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:22.004914 (+ 512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.004916 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.004916 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.004920 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.004928 (+ 8us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.004931 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.004968 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.005040 (+ 72us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.005061 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260201 03:09:22.005201 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.002304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34907 (local address 127.28.181.2:59007)
0201 03:09:22.002464 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:22.002508 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:22.002915 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:22.003090 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:22.003093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:22.003100 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:22.003371 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:22.003372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.003916 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:22.003917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:22.004242 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:22.004243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.004336 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:22.004825 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:22.004826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:22.004841 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:22.004855 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:22.004980 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:22.004983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:22.004985 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:22.005030 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:09:22.005079 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260201 03:09:22.005615 7146 heartbeater.cc:344] Connected to a master server at 127.28.181.62:34907
I20260201 03:09:22.005702 7146 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:22.005897 7146 heartbeater.cc:507] Master 127.28.181.62:34907 requested a full tablet report, sending...
I20260201 03:09:22.006237 6827 ts_manager.cc:194] Registered new tserver with Master: 6978b78a2c2c4f718712707c345a235a (127.28.181.2:35507)
I20260201 03:09:22.006569 6827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.181.2:59007
I20260201 03:09:22.007681 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7018
I20260201 03:09:22.007787 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:09:22.007951 29396 external_mini_cluster.cc:1468] Setting key e24a1d1de9c8314d8887636e2f5bb87b
I20260201 03:09:22.008653 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--builtin_ntp_servers=127.28.181.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260201 03:09:22.078714 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:22.078869 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:22.078895 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:22.080098 7149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:22.080170 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:22.080204 7149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:22.081494 7149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:22.081709 7149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:22.081930 7149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:22.083976 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.083981 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.083981 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.084146 7149 server_base.cc:1060] running on GCE node
I20260201 03:09:22.084322 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:22.084502 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:22.085631 7149 hybrid_clock.cc:648] HybridClock initialized: now 1769915362085628 us; error 36 us; skew 500 ppm
I20260201 03:09:22.086617 7149 webserver.cc:492] Webserver started at http://127.28.181.3:35693/ using document root <none> and password file <none>
I20260201 03:09:22.086800 7149 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:22.086853 7149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:22.086966 7149 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:22.087718 7149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "f1d534112fc44b20b9e9d5be769d6deb"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "fdced87e78323a579a6d3170b4427c7c"
server_key_iv: "9d2c1386d2dec58c2718ae1727b692df"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.087986 7149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "f1d534112fc44b20b9e9d5be769d6deb"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "fdced87e78323a579a6d3170b4427c7c"
server_key_iv: "9d2c1386d2dec58c2718ae1727b692df"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.088939 7149 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:22.089517 7163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.089660 7149 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:22.089732 7149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "f1d534112fc44b20b9e9d5be769d6deb"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "fdced87e78323a579a6d3170b4427c7c"
server_key_iv: "9d2c1386d2dec58c2718ae1727b692df"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.089818 7149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:22.103003 7149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:22.103191 7149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:22.103296 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:22.103470 7149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:22.103711 7149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:22.103756 7149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.103790 7149 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:22.103811 7149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.108441 7149 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:45553
I20260201 03:09:22.108479 7276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:45553 every 8 connection(s)
I20260201 03:09:22.108698 7149 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:09:22.112141 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7149
I20260201 03:09:22.112151 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.109256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:56319 (local address 127.28.181.62:34907)
0201 03:09:22.109318 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.109320 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.109456 (+ 136us) server_negotiation.cc:408] Connection header received
0201 03:09:22.109925 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.109926 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.109936 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.109984 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.110353 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.110766 (+ 413us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:22.111230 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.111385 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:22.111883 (+ 498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.111886 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.111886 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.111890 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.111908 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.111911 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.111965 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.112057 (+ 92us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.112086 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:09:22.112242 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:09:22.112250 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.109190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34907 (local address 127.28.181.3:56319)
0201 03:09:22.109370 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:22.109389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:22.109845 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:22.110028 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:22.110031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:22.110039 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:22.110277 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:22.110279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.110844 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:22.110844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:22.111160 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:22.111161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:22.111286 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:22.111782 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:22.111783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:22.111798 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:22.111811 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:22.111966 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:22.111970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:22.111972 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:22.112060 (+ 88us) client_negotiation.cc:240] Negotiation successful
0201 03:09:22.112115 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260201 03:09:22.112442 29396 external_mini_cluster.cc:1468] Setting key d7e4f2545218107db0471b5a9e685656
I20260201 03:09:22.112633 7277 heartbeater.cc:344] Connected to a master server at 127.28.181.62:34907
I20260201 03:09:22.112709 7277 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:22.112906 7277 heartbeater.cc:507] Master 127.28.181.62:34907 requested a full tablet report, sending...
I20260201 03:09:22.113186 6827 ts_manager.cc:194] Registered new tserver with Master: f1d534112fc44b20b9e9d5be769d6deb (127.28.181.3:45553)
I20260201 03:09:22.113528 6827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.181.3:56319
I20260201 03:09:22.123948 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:09:22.125447 6866 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:22.125051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.28.181.62:34907)
0201 03:09:22.125110 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.125112 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.125175 (+ 63us) server_negotiation.cc:408] Connection header received
0201 03:09:22.125205 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.125206 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.125213 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.125256 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.125382 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37242: BlockingRecv error: recv got EOF from 127.0.0.1:37242 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:09:22.125473 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6887
I20260201 03:09:22.129956 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7018
I20260201 03:09:22.133916 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7149
I20260201 03:09:22.137275 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 6796
2026-02-01T03:09:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (507 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-02-01T03:09:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:22Z Disabled control of system clock
I20260201 03:09:22.160329 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:46829
--webserver_interface=127.28.181.62
--webserver_port=0
--builtin_ntp_servers=127.28.181.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:09:22.226091 7288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:22.226218 7288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:22.226235 7288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:22.227332 7288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:22.227363 7288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:22.227373 7288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:22.227384 7288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:22.227394 7288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:22.228590 7288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:46829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:22.228729 7288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:22.228858 7288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:22.230716 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.230733 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.230738 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.230901 7288 server_base.cc:1060] running on GCE node
I20260201 03:09:22.231102 7288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:22.231328 7288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:22.232460 7288 hybrid_clock.cc:648] HybridClock initialized: now 1769915362232452 us; error 23 us; skew 500 ppm
I20260201 03:09:22.233368 7288 webserver.cc:492] Webserver started at http://127.28.181.62:36821/ using document root <none> and password file <none>
I20260201 03:09:22.233537 7288 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:22.233588 7288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:22.233690 7288 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:22.234464 7288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "21c7ba268fa34af8a4652f56cf26cb49"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "b7ecbd717172547fbe6418f124d96059"
server_key_iv: "e9b4ba52579c270fdbdac2f29a8849be"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.234733 7288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "21c7ba268fa34af8a4652f56cf26cb49"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "b7ecbd717172547fbe6418f124d96059"
server_key_iv: "e9b4ba52579c270fdbdac2f29a8849be"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.235699 7288 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:22.236260 7302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.236399 7288 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:22.236471 7288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "21c7ba268fa34af8a4652f56cf26cb49"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "b7ecbd717172547fbe6418f124d96059"
server_key_iv: "e9b4ba52579c270fdbdac2f29a8849be"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.236536 7288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:22.239281 7288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:22.239460 7288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:22.239563 7288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:22.242646 7288 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:46829
I20260201 03:09:22.242669 7354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:46829 every 8 connection(s)
I20260201 03:09:22.242947 7288 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:22.243434 7355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:22.245250 7355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49: Bootstrap starting.
I20260201 03:09:22.245692 7355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:22.245929 7355 log.cc:826] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:22.246491 7355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49: No bootstrap required, opened a new log
I20260201 03:09:22.247413 7355 raft_consensus.cc:359] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } }
I20260201 03:09:22.247504 7355 raft_consensus.cc:385] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:22.247521 7355 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7ba268fa34af8a4652f56cf26cb49, State: Initialized, Role: FOLLOWER
I20260201 03:09:22.247589 7355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } }
I20260201 03:09:22.247653 7355 raft_consensus.cc:399] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:22.247684 7355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:22.247745 7355 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:22.248298 7355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } }
I20260201 03:09:22.248413 7355 leader_election.cc:304] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c7ba268fa34af8a4652f56cf26cb49; no voters:
I20260201 03:09:22.248554 7355 leader_election.cc:290] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:09:22.248600 7358 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:22.248735 7355 sys_catalog.cc:565] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:22.248734 7358 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [term 1 LEADER]: Becoming Leader. State: Replica: 21c7ba268fa34af8a4652f56cf26cb49, State: Running, Role: LEADER
I20260201 03:09:22.248831 7358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } }
I20260201 03:09:22.249140 7360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c7ba268fa34af8a4652f56cf26cb49. Latest consensus state: current_term: 1 leader_uuid: "21c7ba268fa34af8a4652f56cf26cb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } } }
I20260201 03:09:22.249192 7360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:22.249117 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c7ba268fa34af8a4652f56cf26cb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7ba268fa34af8a4652f56cf26cb49" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 46829 } } }
I20260201 03:09:22.249334 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:22.249393 7364 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:22.249859 7364 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:09:22.251039 7364 catalog_manager.cc:1357] Generated new cluster ID: 019848c4358e4537970e36ebbc8e7e5d
I20260201 03:09:22.251076 7364 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:09:22.253078 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7288
I20260201 03:09:22.253162 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:09:22.253307 29396 external_mini_cluster.cc:1468] Setting key 9dc6975b5b587e55944e32db0ef34a73
I20260201 03:09:22.254218 7364 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:09:22.254335 7364 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:22.255216 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.253685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.28.181.62:46829)
0201 03:09:22.253863 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.253870 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.253907 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:09:22.254028 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.254032 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.254064 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.254173 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.254802 (+ 629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.254804 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.254804 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.254807 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.254827 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.254833 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.254899 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.254951 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.255085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260201 03:09:22.259334 7364 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 21c7ba268fa34af8a4652f56cf26cb49: Generated new TSK 0
I20260201 03:09:22.259477 7364 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:09:22.264344 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--builtin_ntp_servers=127.28.181.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:09:22.331159 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:22.331341 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:22.331374 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:22.332679 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:22.332734 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:22.332767 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:22.334137 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:22.334375 7379 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:22.334569 7379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:22.336733 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.336710 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.336722 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.337172 7379 server_base.cc:1060] running on GCE node
I20260201 03:09:22.337313 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:22.337500 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:22.338632 7379 hybrid_clock.cc:648] HybridClock initialized: now 1769915362338622 us; error 26 us; skew 500 ppm
I20260201 03:09:22.339628 7379 webserver.cc:492] Webserver started at http://127.28.181.1:37221/ using document root <none> and password file <none>
I20260201 03:09:22.339847 7379 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:22.339897 7379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:22.340003 7379 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:22.340797 7379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "5507df37e7014502859255cd9d8bda12"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "a5812249b246632d179303d1ae0eb76b"
server_key_iv: "12c1b27a049259f4bf81a3951f7be41a"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.341063 7379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "5507df37e7014502859255cd9d8bda12"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "a5812249b246632d179303d1ae0eb76b"
server_key_iv: "12c1b27a049259f4bf81a3951f7be41a"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.342116 7379 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:22.342734 7393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.342873 7379 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:22.342932 7379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "5507df37e7014502859255cd9d8bda12"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "a5812249b246632d179303d1ae0eb76b"
server_key_iv: "12c1b27a049259f4bf81a3951f7be41a"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.342998 7379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:22.345736 7379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:22.345945 7379 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:22.346040 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:22.346262 7379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:22.346553 7379 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:22.346591 7379 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.346609 7379 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:22.346621 7379 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.352761 7379 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:40225
I20260201 03:09:22.352840 7506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:40225 every 8 connection(s)
I20260201 03:09:22.353042 7379 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
I20260201 03:09:22.355401 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.353715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:55667 (local address 127.28.181.62:46829)
0201 03:09:22.353822 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.353824 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.353976 (+ 152us) server_negotiation.cc:408] Connection header received
0201 03:09:22.354456 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.354457 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.354461 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.354510 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.355159 (+ 649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.355160 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.355160 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.355164 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.355176 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.355179 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.355219 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.355344 (+ 125us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.355356 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:09:22.355611 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.353669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46829 (local address 127.28.181.1:55667)
0201 03:09:22.353877 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:22.353902 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:22.354374 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:22.354563 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:22.354567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:22.354576 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:22.355030 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:22.355031 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:22.355056 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:22.355079 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:22.355269 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:22.355274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:22.355276 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:22.355371 (+ 95us) client_negotiation.cc:240] Negotiation successful
0201 03:09:22.355414 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260201 03:09:22.356076 7507 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46829
I20260201 03:09:22.356158 7507 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:22.356284 7507 heartbeater.cc:507] Master 127.28.181.62:46829 requested a full tablet report, sending...
I20260201 03:09:22.356673 7319 ts_manager.cc:194] Registered new tserver with Master: 5507df37e7014502859255cd9d8bda12 (127.28.181.1:40225)
I20260201 03:09:22.357831 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7379
I20260201 03:09:22.357926 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:09:22.358094 29396 external_mini_cluster.cc:1468] Setting key 8fab0863986c49073db929fb84249d41
I20260201 03:09:22.358932 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--builtin_ntp_servers=127.28.181.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:09:22.423837 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:22.423962 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:22.423978 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:22.425141 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:22.425179 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:22.425199 7510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:22.426465 7510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:22.426658 7510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:22.426846 7510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:22.428889 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.428980 7510 server_base.cc:1060] running on GCE node
W20260201 03:09:22.428889 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.428980 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.429217 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:22.429392 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:22.430541 7510 hybrid_clock.cc:648] HybridClock initialized: now 1769915362430522 us; error 30 us; skew 500 ppm
I20260201 03:09:22.431461 7510 webserver.cc:492] Webserver started at http://127.28.181.2:42347/ using document root <none> and password file <none>
I20260201 03:09:22.431634 7510 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:22.431690 7510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:22.431808 7510 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:22.432539 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "722e95e024f74d8d9c5de9f9d01c2295"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "2215ea97b2c51ff2826555160578d367"
server_key_iv: "654548ada6666d78750c5efa04f2e784"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.432801 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "722e95e024f74d8d9c5de9f9d01c2295"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "2215ea97b2c51ff2826555160578d367"
server_key_iv: "654548ada6666d78750c5efa04f2e784"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.433818 7510 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260201 03:09:22.434422 7524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.434568 7510 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:22.434639 7510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "722e95e024f74d8d9c5de9f9d01c2295"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "2215ea97b2c51ff2826555160578d367"
server_key_iv: "654548ada6666d78750c5efa04f2e784"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.434701 7510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:22.437414 7510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:22.437582 7510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:22.437674 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:22.437840 7510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:22.438089 7510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:22.438133 7510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.438169 7510 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:22.438189 7510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.443171 7510 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:42403
I20260201 03:09:22.443203 7637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:42403 every 8 connection(s)
I20260201 03:09:22.443428 7510 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:09:22.445209 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.443984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:46987 (local address 127.28.181.62:46829)
0201 03:09:22.444024 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.444026 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.444188 (+ 162us) server_negotiation.cc:408] Connection header received
0201 03:09:22.444493 (+ 305us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.444494 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.444500 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.444545 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.445040 (+ 495us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.445041 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.445041 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.445044 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.445058 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.445061 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.445111 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.445150 (+ 39us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.445162 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:09:22.445358 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.443940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46829 (local address 127.28.181.2:46987)
0201 03:09:22.444101 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:22.444120 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:22.444432 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:22.444576 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:22.444578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:22.444585 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:22.444949 (+ 364us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:22.444950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:22.444964 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:22.444978 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:22.445110 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:22.445112 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:22.445114 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:22.445168 (+ 54us) client_negotiation.cc:240] Negotiation successful
0201 03:09:22.445196 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":49,"threads_started":1}
I20260201 03:09:22.445860 7638 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46829
I20260201 03:09:22.445930 7638 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:22.446072 7638 heartbeater.cc:507] Master 127.28.181.62:46829 requested a full tablet report, sending...
I20260201 03:09:22.446382 7319 ts_manager.cc:194] Registered new tserver with Master: 722e95e024f74d8d9c5de9f9d01c2295 (127.28.181.2:42403)
I20260201 03:09:22.452752 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7510
I20260201 03:09:22.452826 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:09:22.452941 29396 external_mini_cluster.cc:1468] Setting key 083fc0bd98ef35d8a84f7f3c2f52f94d
I20260201 03:09:22.458125 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--builtin_ntp_servers=127.28.181.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:09:22.526472 7641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:22.526602 7641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:22.526620 7641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:22.527855 7641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:22.527894 7641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:22.527915 7641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:22.529291 7641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:22.529465 7641 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:22.529686 7641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:22.531762 7649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.531883 7647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:22.531795 7646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:22.532109 7641 server_base.cc:1060] running on GCE node
I20260201 03:09:22.532259 7641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:22.532446 7641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:22.533572 7641 hybrid_clock.cc:648] HybridClock initialized: now 1769915362533557 us; error 31 us; skew 500 ppm
I20260201 03:09:22.534543 7641 webserver.cc:492] Webserver started at http://127.28.181.3:33571/ using document root <none> and password file <none>
I20260201 03:09:22.534736 7641 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:22.534773 7641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:22.534898 7641 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:22.535727 7641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "b32190b324644a2cb28b6fa5fb74ce0c"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "1d7b1626f37ff6c9f99f76c13ec58488"
server_key_iv: "a7a1c03cf712ef2711a81902b181be0f"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.536019 7641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "b32190b324644a2cb28b6fa5fb74ce0c"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "1d7b1626f37ff6c9f99f76c13ec58488"
server_key_iv: "a7a1c03cf712ef2711a81902b181be0f"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.537051 7641 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:22.537720 7655 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.537901 7641 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:22.537976 7641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "b32190b324644a2cb28b6fa5fb74ce0c"
format_stamp: "Formatted at 2026-02-01 03:09:22 on dist-test-slave-ck1n"
server_key: "1d7b1626f37ff6c9f99f76c13ec58488"
server_key_iv: "a7a1c03cf712ef2711a81902b181be0f"
server_key_version: "encryptionkey@0"
I20260201 03:09:22.538041 7641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:22.541100 7641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:22.541266 7641 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:22.541366 7641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:22.541539 7641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:22.541817 7641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:22.541853 7641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.541887 7641 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:22.541911 7641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:22.546885 7641 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:45307
I20260201 03:09:22.546926 7768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:45307 every 8 connection(s)
I20260201 03:09:22.547142 7641 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:09:22.549207 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.547629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:41175 (local address 127.28.181.62:46829)
0201 03:09:22.547702 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.547704 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.547856 (+ 152us) server_negotiation.cc:408] Connection header received
0201 03:09:22.548233 (+ 377us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.548234 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.548242 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.548291 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.548978 (+ 687us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:22.548979 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:22.548979 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:09:22.548984 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:22.548996 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:09:22.548999 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:22.549058 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:22.549129 (+ 71us) server_negotiation.cc:300] Negotiation successful
0201 03:09:22.549145 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:09:22.549408 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:22.547604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:46829 (local address 127.28.181.3:41175)
0201 03:09:22.547771 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:22.547798 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:22.548142 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:22.548340 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:22.548342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:22.548349 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:22.548881 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:09:22.548881 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:22.548896 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:09:22.548910 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:22.549055 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:22.549058 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:22.549060 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:22.549154 (+ 94us) client_negotiation.cc:240] Negotiation successful
0201 03:09:22.549191 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260201 03:09:22.549829 7769 heartbeater.cc:344] Connected to a master server at 127.28.181.62:46829
I20260201 03:09:22.549914 7769 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:22.550040 7769 heartbeater.cc:507] Master 127.28.181.62:46829 requested a full tablet report, sending...
I20260201 03:09:22.550309 7319 ts_manager.cc:194] Registered new tserver with Master: b32190b324644a2cb28b6fa5fb74ce0c (127.28.181.3:45307)
I20260201 03:09:22.552107 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7641
I20260201 03:09:22.552188 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:09:22.552302 29396 external_mini_cluster.cc:1468] Setting key 37513c0cd955dce3d3b55ceb14efaea2
I20260201 03:09:22.552870 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:09:22.554318 7378 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:09:22.553977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.28.181.62:46829)
0201 03:09:22.554033 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:22.554035 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:22.554079 (+ 44us) server_negotiation.cc:408] Connection header received
0201 03:09:22.554123 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:22.554124 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:22.554128 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:22.554167 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:22.554279 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55524: BlockingRecv error: recv got EOF from 127.0.0.1:55524 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260201 03:09:22.554433 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7379
I20260201 03:09:22.559607 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7510
I20260201 03:09:22.563725 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7641
I20260201 03:09:22.567497 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7288
2026-02-01T03:09:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (430 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:09:22 dist-test-slave-ck1n krb5kdc[7779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:09:22 dist-test-slave-ck1n krb5kdc[7779](info): set up 2 sockets
Feb 01 03:09:22 dist-test-slave-ck1n krb5kdc[7779](info): commencing operation
krb5kdc: starting...
W20260201 03:09:24.274051 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.688s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:09:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:24.380016 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38789
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:38789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:24.445660 7795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:24.445849 7795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:24.445874 7795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:24.447070 7795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:09:24.447113 7795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:24.447124 7795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:24.447135 7795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:09:24.447184 7795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:09:24.448479 7795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:38789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:24.448618 7795 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:24.448783 7795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:24.450867 7800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.450865 7803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:24.451083 7795 server_base.cc:1060] running on GCE node
W20260201 03:09:24.450867 7801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:24.451335 7795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:24.451537 7795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:24.452674 7795 hybrid_clock.cc:648] HybridClock initialized: now 1769915364452663 us; error 21 us; skew 500 ppm
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:24.454636 7795 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:09:24.455099 7795 webserver.cc:492] Webserver started at http://127.28.181.62:35061/ using document root <none> and password file <none>
I20260201 03:09:24.455267 7795 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:24.455298 7795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:24.455411 7795 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:24.456169 7795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "2709a2e1b90049bcb595ed81fb3695f3"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "45c293993a03bb077887d7ad02522102"
server_key_iv: "9b619423027c061c55b324c6351c2010"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.456446 7795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "2709a2e1b90049bcb595ed81fb3695f3"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "45c293993a03bb077887d7ad02522102"
server_key_iv: "9b619423027c061c55b324c6351c2010"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.457427 7795 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:24.458004 7810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.458160 7795 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:24.458236 7795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "2709a2e1b90049bcb595ed81fb3695f3"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "45c293993a03bb077887d7ad02522102"
server_key_iv: "9b619423027c061c55b324c6351c2010"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.458303 7795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:24.477864 7795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:24.482085 7795 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:24.482215 7795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:24.485556 7795 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:38789
I20260201 03:09:24.485582 7862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:38789 every 8 connection(s)
I20260201 03:09:24.485859 7795 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:09:24.486361 7863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:24.488299 7863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: Bootstrap starting.
I20260201 03:09:24.488790 7863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:24.489012 7863 log.cc:826] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:24.489570 7863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: No bootstrap required, opened a new log
I20260201 03:09:24.490598 7863 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } }
I20260201 03:09:24.490694 7863 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:24.490711 7863 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2709a2e1b90049bcb595ed81fb3695f3, State: Initialized, Role: FOLLOWER
I20260201 03:09:24.490836 7863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } }
I20260201 03:09:24.490896 7863 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:09:24.490943 7863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:09:24.490985 7863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:24.491413 7863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } }
I20260201 03:09:24.491513 7863 leader_election.cc:304] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2709a2e1b90049bcb595ed81fb3695f3; no voters:
I20260201 03:09:24.491657 7863 leader_election.cc:290] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:09:24.491703 7866 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:24.491855 7866 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [term 1 LEADER]: Becoming Leader. State: Replica: 2709a2e1b90049bcb595ed81fb3695f3, State: Running, Role: LEADER
I20260201 03:09:24.491842 7863 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:09:24.491945 7866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } }
I20260201 03:09:24.492249 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2709a2e1b90049bcb595ed81fb3695f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } } }
I20260201 03:09:24.492378 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:24.492802 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2709a2e1b90049bcb595ed81fb3695f3. Latest consensus state: current_term: 1 leader_uuid: "2709a2e1b90049bcb595ed81fb3695f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2709a2e1b90049bcb595ed81fb3695f3" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 38789 } } }
I20260201 03:09:24.492926 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3 [sys.catalog]: This master's current role is: LEADER
I20260201 03:09:24.493008 7873 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:09:24.493618 7873 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:09:24.493829 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7795
I20260201 03:09:24.493922 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:09:24.494091 29396 external_mini_cluster.cc:1468] Setting key 6fe8b9b31029912d52adfd8728780b28
I20260201 03:09:24.495137 7873 catalog_manager.cc:1357] Generated new cluster ID: 1667f9cdd1434035b927a07cea49734f
I20260201 03:09:24.495213 7873 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:24.506754 7873 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:09:24.506920 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.494624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.28.181.62:38789)
0201 03:09:24.498105 (+ 3481us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.498119 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.498165 (+ 46us) server_negotiation.cc:408] Connection header received
0201 03:09:24.498279 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.498284 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.498341 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.498437 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:24.499342 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.500453 (+ 1111us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.501293 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.501490 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.503484 (+ 1994us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:24.503489 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:24.503489 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:24.503505 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:24.505130 (+ 1625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.505455 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.505458 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.505458 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.505511 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.505649 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.505650 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.505650 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.505885 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:24.506019 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.506645 (+ 626us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.506762 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3448,"thread_start_us":68,"threads_started":1}
I20260201 03:09:24.507395 7873 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:09:24.524617 7873 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: Generated new TSK 0
I20260201 03:09:24.524765 7873 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:24.562436 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38789
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:24.629639 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:24.629894 7891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:24.629941 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:24.631145 7891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:24.631204 7891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:24.631242 7891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:09:24.632539 7891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:24.632755 7891 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:24.632912 7891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:24.634824 7896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.634829 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.634944 7899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:24.635151 7891 server_base.cc:1060] running on GCE node
I20260201 03:09:24.635337 7891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:24.635536 7891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:24.636674 7891 hybrid_clock.cc:648] HybridClock initialized: now 1769915364636652 us; error 35 us; skew 500 ppm
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:24.638528 7891 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:09:24.638996 7891 webserver.cc:492] Webserver started at http://127.28.181.1:34847/ using document root <none> and password file <none>
I20260201 03:09:24.639173 7891 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:24.639220 7891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:24.639335 7891 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:24.640107 7891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "75f3111db4364119a05b8d79e606c03c"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "4d96699d05cbcf4f27b584a4194bbd06"
server_key_iv: "a53ff8352674d39e9ae1e96bb5f4082f"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.640381 7891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "75f3111db4364119a05b8d79e606c03c"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "4d96699d05cbcf4f27b584a4194bbd06"
server_key_iv: "a53ff8352674d39e9ae1e96bb5f4082f"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.641368 7891 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260201 03:09:24.641943 7906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.642097 7891 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260201 03:09:24.642171 7891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "75f3111db4364119a05b8d79e606c03c"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "4d96699d05cbcf4f27b584a4194bbd06"
server_key_iv: "a53ff8352674d39e9ae1e96bb5f4082f"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.642238 7891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:24.659543 7891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:24.660851 7891 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:24.660987 7891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:24.661226 7891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:24.661509 7891 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:24.661554 7891 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.661582 7891 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:24.661607 7891 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.666612 7891 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:43257
I20260201 03:09:24.666649 8019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:43257 every 8 connection(s)
I20260201 03:09:24.666875 7891 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:24.673231 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.667358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:49753 (local address 127.28.181.62:38789)
0201 03:09:24.667402 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.667404 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.667551 (+ 147us) server_negotiation.cc:408] Connection header received
0201 03:09:24.667923 (+ 372us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.667924 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.667935 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.667982 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:24.668567 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.668974 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.669416 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.669509 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.671371 (+ 1862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:24.671375 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:24.671375 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:24.671381 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:24.672371 (+ 990us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.672648 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.672649 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.672649 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.672668 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.672806 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.672807 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.672807 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.672874 (+ 67us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:24.672911 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.673110 (+ 199us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.673173 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260201 03:09:24.673275 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.667317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38789 (local address 127.28.181.1:49753)
0201 03:09:24.667449 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.667476 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.667860 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.668034 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.668037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.668267 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:24.668504 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.668507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.669043 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.669043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.669359 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.669360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.669442 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.670158 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:24.670171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:24.671300 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:24.672430 (+ 1130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.672435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.672436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.672594 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.672714 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.672714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.672715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.672767 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.672914 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:24.672917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:24.672997 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.673083 (+ 86us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.673164 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260201 03:09:24.673646 8020 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38789
I20260201 03:09:24.673708 8020 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:24.673904 8020 heartbeater.cc:507] Master 127.28.181.62:38789 requested a full tablet report, sending...
I20260201 03:09:24.674294 7827 ts_manager.cc:194] Registered new tserver with Master: 75f3111db4364119a05b8d79e606c03c (127.28.181.1:43257)
I20260201 03:09:24.674806 7827 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:49753
I20260201 03:09:24.675658 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 7891
I20260201 03:09:24.675753 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:09:24.675911 29396 external_mini_cluster.cc:1468] Setting key 67bc43b72fe1e5650d9fae8e3361972c
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:24.709811 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38789
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:24.778745 8027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:24.778892 8027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:24.778913 8027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:24.780190 8027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:24.780230 8027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:24.780284 8027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:09:24.781682 8027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:24.781893 8027 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:24.782049 8027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:24.783973 8033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.783978 8035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.784044 8032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:24.784119 8027 server_base.cc:1060] running on GCE node
I20260201 03:09:24.784250 8027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:24.784461 8027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:24.785589 8027 hybrid_clock.cc:648] HybridClock initialized: now 1769915364785572 us; error 36 us; skew 500 ppm
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:24.787580 8027 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:09:24.788035 8027 webserver.cc:492] Webserver started at http://127.28.181.2:40687/ using document root <none> and password file <none>
I20260201 03:09:24.788224 8027 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:24.788259 8027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:24.788378 8027 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:24.789186 8027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "4c68406c7093460f99338c3e563efd4d"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "9edb0008ddac97747a1e0ce25ba58ca3"
server_key_iv: "bc4b8932320764eda8e675fa5db5caed"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.789469 8027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "4c68406c7093460f99338c3e563efd4d"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "9edb0008ddac97747a1e0ce25ba58ca3"
server_key_iv: "bc4b8932320764eda8e675fa5db5caed"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.790632 8027 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:09:24.791229 8042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.791388 8027 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:24.791462 8027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "4c68406c7093460f99338c3e563efd4d"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "9edb0008ddac97747a1e0ce25ba58ca3"
server_key_iv: "bc4b8932320764eda8e675fa5db5caed"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.791530 8027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:24.809020 8027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:24.810461 8027 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:24.810596 8027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:24.810765 8027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:24.811040 8027 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:24.811089 8027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.811117 8027 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:24.811157 8027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.816056 8027 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:36055
I20260201 03:09:24.816093 8155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:36055 every 8 connection(s)
I20260201 03:09:24.816318 8027 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:24.822590 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.816868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:38965 (local address 127.28.181.62:38789)
0201 03:09:24.816934 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.816936 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.817040 (+ 104us) server_negotiation.cc:408] Connection header received
0201 03:09:24.817386 (+ 346us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.817387 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.817397 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.817438 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:24.818052 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.818472 (+ 420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.818896 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.818989 (+ 93us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.820756 (+ 1767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:24.820759 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:24.820760 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:24.820766 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:24.821729 (+ 963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.822035 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.822036 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.822036 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.822054 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.822192 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.822193 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.822193 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.822259 (+ 66us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:24.822297 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.822477 (+ 180us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.822535 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:09:24.822674 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.816819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38789 (local address 127.28.181.2:38965)
0201 03:09:24.816940 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.816967 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.817322 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.817479 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.817481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.817703 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:24.817985 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.817988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.818535 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.818536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.818837 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.818838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.818918 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.819604 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:24.819616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:24.820685 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:24.821826 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.821830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.821832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.821981 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.822101 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.822102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.822102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.822153 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.822297 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:24.822300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:24.822379 (+ 79us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.822454 (+ 75us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.822531 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":75,"threads_started":1}
I20260201 03:09:24.823028 8156 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38789
I20260201 03:09:24.823091 8156 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:24.823287 8156 heartbeater.cc:507] Master 127.28.181.62:38789 requested a full tablet report, sending...
I20260201 03:09:24.823359 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 8027
I20260201 03:09:24.823453 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:09:24.823628 29396 external_mini_cluster.cc:1468] Setting key b4f12a22f786bd5e503426c8718fa689
I20260201 03:09:24.823645 7827 ts_manager.cc:194] Registered new tserver with Master: 4c68406c7093460f99338c3e563efd4d (127.28.181.2:36055)
I20260201 03:09:24.824011 7827 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.2@KRBTEST.COM'} at 127.28.181.2:38965
WARNING: no policy specified for kudu/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:09:24.857719 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.3:0
--local_ip_for_outbound_sockets=127.28.181.3
--webserver_interface=127.28.181.3
--webserver_port=0
--tserver_master_addrs=127.28.181.62:38789
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:09:24.923408 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:09:24.923542 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:09:24.923559 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:09:24.924750 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:09:24.924786 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:09:24.924808 8163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.3
I20260201 03:09:24.926132 8163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.181.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:38789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.3
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:09:24.926270 8163 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:09:24.926438 8163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:09:24.928308 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.928315 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:09:24.928323 8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:09:24.928561 8163 server_base.cc:1060] running on GCE node
I20260201 03:09:24.928691 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:09:24.928867 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:09:24.929996 8163 hybrid_clock.cc:648] HybridClock initialized: now 1769915364929981 us; error 26 us; skew 500 ppm
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:09:24.931886 8163 init.cc:377] Logged in from keytab as kudu/127.28.181.3@KRBTEST.COM (short username kudu)
I20260201 03:09:24.932307 8163 webserver.cc:492] Webserver started at http://127.28.181.3:40551/ using document root <none> and password file <none>
I20260201 03:09:24.932466 8163 fs_manager.cc:362] Metadata directory not provided
I20260201 03:09:24.932511 8163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:09:24.932614 8163 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:09:24.933372 8163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/instance:
uuid: "af75e546246b4ee0b7e324b99e308142"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "cbac8ba23b2021ab49cd98c860e4c7bb"
server_key_iv: "23838b5c8c45e3c47f8b6829c44e937e"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.933630 8163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance:
uuid: "af75e546246b4ee0b7e324b99e308142"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "cbac8ba23b2021ab49cd98c860e4c7bb"
server_key_iv: "23838b5c8c45e3c47f8b6829c44e937e"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.934633 8163 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:09:24.935177 8178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.935318 8163 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260201 03:09:24.935388 8163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
uuid: "af75e546246b4ee0b7e324b99e308142"
format_stamp: "Formatted at 2026-02-01 03:09:24 on dist-test-slave-ck1n"
server_key: "cbac8ba23b2021ab49cd98c860e4c7bb"
server_key_iv: "23838b5c8c45e3c47f8b6829c44e937e"
server_key_version: "encryptionkey@0"
I20260201 03:09:24.935451 8163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:09:24.942912 8163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:09:24.944424 8163 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:09:24.944540 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:09:24.944717 8163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:09:24.944950 8163 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:09:24.944993 8163 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.945026 8163 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:09:24.945051 8163 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:09:24.949671 8163 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.3:36295
I20260201 03:09:24.949714 8291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.3:36295 every 8 connection(s)
I20260201 03:09:24.949946 8163 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/data/info.pb
I20260201 03:09:24.950628 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 8163
I20260201 03:09:24.950721 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/instance
I20260201 03:09:24.950870 29396 external_mini_cluster.cc:1468] Setting key e186a188110a0b8163e7b2e24aceed91
Feb 01 03:09:24 dist-test-slave-ck1n krb5kdc[7779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915364, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.3@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:09:24.956738 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.950538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.3:44977 (local address 127.28.181.62:38789)
0201 03:09:24.950599 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.950601 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.950729 (+ 128us) server_negotiation.cc:408] Connection header received
0201 03:09:24.951198 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.951199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.951210 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.951258 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:24.951825 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.952251 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.952702 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.952858 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.954627 (+ 1769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:24.954633 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:24.954634 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:24.954639 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:24.955683 (+ 1044us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.955997 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.955998 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.955998 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.956020 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.956214 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.956215 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.956215 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.956303 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:24.956368 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.956583 (+ 215us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.956661 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:09:24.956818 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.950490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:38789 (local address 127.28.181.3:44977)
0201 03:09:24.950650 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.950673 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.951107 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.951306 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.951309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.951533 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:09:24.951746 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.951749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.952326 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.952327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.952632 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.952633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.952747 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.953386 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:09:24.953398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:09:24.954545 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:09:24.955757 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.955761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.955763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.955924 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.956090 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:09:24.956090 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:09:24.956090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:09:24.956160 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:09:24.956358 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:09:24.956361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:09:24.956467 (+ 106us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.956567 (+ 100us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.956666 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260201 03:09:24.957271 8292 heartbeater.cc:344] Connected to a master server at 127.28.181.62:38789
I20260201 03:09:24.957338 8292 heartbeater.cc:461] Registering TS with master...
I20260201 03:09:24.957458 8292 heartbeater.cc:507] Master 127.28.181.62:38789 requested a full tablet report, sending...
I20260201 03:09:24.957705 7826 ts_manager.cc:194] Registered new tserver with Master: af75e546246b4ee0b7e324b99e308142 (127.28.181.3:36295)
I20260201 03:09:24.958079 7826 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.3@KRBTEST.COM'} at 127.28.181.3:44977
I20260201 03:09:24.962208 29396 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:09:24.967486 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.963292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.28.181.62:38789)
0201 03:09:24.963351 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.963353 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.963431 (+ 78us) server_negotiation.cc:408] Connection header received
0201 03:09:24.963511 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.963511 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.963519 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.963577 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:09:24.964135 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.964498 (+ 363us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.964966 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.965128 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.965802 (+ 674us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:09:24.965804 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:09:24.965804 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:09:24.965810 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:09:24.966666 (+ 856us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.966934 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.966934 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.966935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.966957 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:09:24.967089 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:09:24.967090 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:09:24.967090 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:09:24.967169 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:09:24.967259 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.967343 (+ 84us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.967407 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:09:24.968582 7826 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:09:24.969540 7826 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:09:24.973850 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36055 (local address 127.0.0.1:35664)
0201 03:09:24.971281 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.971304 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.971372 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.971702 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.971703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.971704 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:24.971924 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.971925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.973046 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.973047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.973542 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.973544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.973640 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.973671 (+ 31us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.973711 (+ 40us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.973748 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260201 03:09:24.973935 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:36295 (local address 127.0.0.1:39656)
0201 03:09:24.971454 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.971466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.971506 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.971700 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.971702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.971703 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:24.971926 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.971927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.973003 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.973004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.973726 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.973727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.973775 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.973777 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.973826 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.973863 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"spinlock_wait_cycles":1664,"thread_start_us":55,"threads_started":1}
I20260201 03:09:24.973861 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:43257 (local address 127.0.0.1:50364)
0201 03:09:24.971446 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:24.971454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:24.971502 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:24.971922 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:24.971923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:24.971924 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:24.972118 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.972120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.972881 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.972882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:24.973622 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:24.973624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.973681 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.973684 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:24.973713 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:09:24.973753 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
I20260201 03:09:24.974558 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.28.181.2:36055)
0201 03:09:24.971574 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.971580 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.971589 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:09:24.971618 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.971620 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.971636 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.971694 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:24.972004 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.972962 (+ 958us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.973663 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.974434 (+ 771us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.974462 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.974484 (+ 22us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.974510 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":81,"threads_started":1}
I20260201 03:09:24.974632 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.28.181.3:36295)
0201 03:09:24.971561 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.971568 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.971579 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:09:24.971605 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.971606 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.971619 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.971686 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:24.972001 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.972929 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.974034 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.974480 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.974517 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.974541 (+ 24us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.974571 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260201 03:09:24.975054 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:24.971425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.28.181.1:43257)
0201 03:09:24.971796 (+ 371us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:24.971801 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:24.971807 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:09:24.971832 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:24.971833 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:24.971844 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:24.971908 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:24.972192 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.972793 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:24.974240 (+ 1447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:24.974857 (+ 617us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:24.974892 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:24.974918 (+ 26us) server_negotiation.cc:300] Negotiation successful
0201 03:09:24.974965 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":63,"threads_started":1}
I20260201 03:09:24.975040 8226 tablet_service.cc:1505] Processing CreateTablet for tablet c9068648263b491b8aff8f36bceca218 (DEFAULT_TABLE table=test-table [id=791b49cc4e97462eb3f842d6a7b584bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:24.975064 8090 tablet_service.cc:1505] Processing CreateTablet for tablet c9068648263b491b8aff8f36bceca218 (DEFAULT_TABLE table=test-table [id=791b49cc4e97462eb3f842d6a7b584bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:24.975274 8226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9068648263b491b8aff8f36bceca218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:24.975306 8090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9068648263b491b8aff8f36bceca218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:24.975437 7954 tablet_service.cc:1505] Processing CreateTablet for tablet c9068648263b491b8aff8f36bceca218 (DEFAULT_TABLE table=test-table [id=791b49cc4e97462eb3f842d6a7b584bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:09:24.975648 7954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9068648263b491b8aff8f36bceca218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:09:24.977119 8306 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Bootstrap starting.
I20260201 03:09:24.977643 8307 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: Bootstrap starting.
I20260201 03:09:24.977766 8306 tablet_bootstrap.cc:654] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:24.978117 8306 log.cc:826] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:24.978176 8307 tablet_bootstrap.cc:654] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:24.978410 8307 log.cc:826] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:24.978659 8308 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: Bootstrap starting.
I20260201 03:09:24.978734 8306 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: No bootstrap required, opened a new log
I20260201 03:09:24.978787 8306 ts_tablet_manager.cc:1403] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:24.978945 8307 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: No bootstrap required, opened a new log
I20260201 03:09:24.979008 8307 ts_tablet_manager.cc:1403] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:09:24.979171 8308 tablet_bootstrap.cc:654] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: Neither blocks nor log segments found. Creating new log.
I20260201 03:09:24.979429 8308 log.cc:826] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: Log is configured to *not* fsync() on all Append() calls
I20260201 03:09:24.980082 8308 tablet_bootstrap.cc:492] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: No bootstrap required, opened a new log
I20260201 03:09:24.980147 8308 ts_tablet_manager.cc:1403] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:09:24.980290 8307 raft_consensus.cc:359] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.980398 8307 raft_consensus.cc:385] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:24.980430 8307 raft_consensus.cc:740] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c68406c7093460f99338c3e563efd4d, State: Initialized, Role: FOLLOWER
I20260201 03:09:24.980389 8306 raft_consensus.cc:359] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.980504 8306 raft_consensus.cc:385] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:24.980532 8306 raft_consensus.cc:740] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75e546246b4ee0b7e324b99e308142, State: Initialized, Role: FOLLOWER
I20260201 03:09:24.980499 8307 consensus_queue.cc:260] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.980607 8306 consensus_queue.cc:260] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.980720 8307 ts_tablet_manager.cc:1434] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:09:24.980763 8156 heartbeater.cc:499] Master 127.28.181.62:38789 was elected leader, sending a full tablet report...
I20260201 03:09:24.980778 8306 ts_tablet_manager.cc:1434] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:09:24.980859 8292 heartbeater.cc:499] Master 127.28.181.62:38789 was elected leader, sending a full tablet report...
I20260201 03:09:24.981526 8308 raft_consensus.cc:359] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.981650 8308 raft_consensus.cc:385] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:09:24.981676 8308 raft_consensus.cc:740] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f3111db4364119a05b8d79e606c03c, State: Initialized, Role: FOLLOWER
I20260201 03:09:24.981761 8308 consensus_queue.cc:260] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:24.981945 8308 ts_tablet_manager.cc:1434] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260201 03:09:24.981997 8020 heartbeater.cc:499] Master 127.28.181.62:38789 was elected leader, sending a full tablet report...
I20260201 03:09:25.035051 8312 raft_consensus.cc:493] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:09:25.035133 8312 raft_consensus.cc:515] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:25.035344 8312 leader_election.cc:290] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af75e546246b4ee0b7e324b99e308142 (127.28.181.3:36295), 75f3111db4364119a05b8d79e606c03c (127.28.181.1:43257)
I20260201 03:09:25.037549 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.035491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.3:36295 (local address 127.28.181.2:46223)
0201 03:09:25.035535 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:25.035545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:25.035582 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:25.035731 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:25.035732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:25.035735 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:25.035928 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:25.035930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.036771 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:25.036772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:25.037305 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:25.037306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.037404 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:25.037407 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:09:25.037436 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:09:25.037468 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260201 03:09:25.037573 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.035590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:43257 (local address 127.28.181.2:45477)
0201 03:09:25.035712 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:25.035721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0201 03:09:25.035773 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:09:25.035885 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:09:25.035885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:09:25.035886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:09:25.036132 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:25.036133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.036659 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:09:25.036660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:09:25.037305 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:09:25.037307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.037406 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:25.037410 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:09:25.037439 (+ 29us) client_negotiation.cc:240] Negotiation successful
0201 03:09:25.037470 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260201 03:09:25.037932 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.035564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:46223 (local address 127.28.181.3:36295)
0201 03:09:25.035621 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:25.035623 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:25.035632 (+ 9us) server_negotiation.cc:408] Connection header received
0201 03:09:25.035653 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:25.035654 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:25.035662 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:25.035720 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:25.036000 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.036685 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:25.037401 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.037841 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:25.037847 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:25.037866 (+ 19us) server_negotiation.cc:300] Negotiation successful
0201 03:09:25.037886 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:09:25.037978 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.035682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.2:45477 (local address 127.28.181.1:43257)
0201 03:09:25.035716 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:25.035718 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:25.035764 (+ 46us) server_negotiation.cc:408] Connection header received
0201 03:09:25.035819 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:25.035819 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:25.035826 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:25.035874 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:09:25.036200 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.036591 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:25.037404 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.037858 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:25.037865 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:25.037886 (+ 21us) server_negotiation.cc:300] Negotiation successful
0201 03:09:25.037924 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260201 03:09:25.038190 8246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9068648263b491b8aff8f36bceca218" candidate_uuid: "4c68406c7093460f99338c3e563efd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75e546246b4ee0b7e324b99e308142" is_pre_election: true
I20260201 03:09:25.038205 7974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9068648263b491b8aff8f36bceca218" candidate_uuid: "4c68406c7093460f99338c3e563efd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f3111db4364119a05b8d79e606c03c" is_pre_election: true
I20260201 03:09:25.038311 8246 raft_consensus.cc:2468] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c68406c7093460f99338c3e563efd4d in term 0.
I20260201 03:09:25.038326 7974 raft_consensus.cc:2468] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c68406c7093460f99338c3e563efd4d in term 0.
I20260201 03:09:25.038508 8046 leader_election.cc:304] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c68406c7093460f99338c3e563efd4d, af75e546246b4ee0b7e324b99e308142; no voters:
I20260201 03:09:25.038627 8312 raft_consensus.cc:2804] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:09:25.038682 8312 raft_consensus.cc:493] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:09:25.038697 8312 raft_consensus.cc:3060] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:25.039402 8312 raft_consensus.cc:515] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:25.039506 8312 leader_election.cc:290] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [CANDIDATE]: Term 1 election: Requested vote from peers af75e546246b4ee0b7e324b99e308142 (127.28.181.3:36295), 75f3111db4364119a05b8d79e606c03c (127.28.181.1:43257)
I20260201 03:09:25.039752 8246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9068648263b491b8aff8f36bceca218" candidate_uuid: "4c68406c7093460f99338c3e563efd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75e546246b4ee0b7e324b99e308142"
I20260201 03:09:25.039781 7974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9068648263b491b8aff8f36bceca218" candidate_uuid: "4c68406c7093460f99338c3e563efd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f3111db4364119a05b8d79e606c03c"
I20260201 03:09:25.039829 8246 raft_consensus.cc:3060] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:25.039827 7974 raft_consensus.cc:3060] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:09:25.040401 8246 raft_consensus.cc:2468] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c68406c7093460f99338c3e563efd4d in term 1.
I20260201 03:09:25.040407 7974 raft_consensus.cc:2468] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c68406c7093460f99338c3e563efd4d in term 1.
I20260201 03:09:25.040580 8046 leader_election.cc:304] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c68406c7093460f99338c3e563efd4d, af75e546246b4ee0b7e324b99e308142; no voters:
I20260201 03:09:25.040686 8312 raft_consensus.cc:2804] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:09:25.040830 8312 raft_consensus.cc:697] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 1 LEADER]: Becoming Leader. State: Replica: 4c68406c7093460f99338c3e563efd4d, State: Running, Role: LEADER
I20260201 03:09:25.040928 8312 consensus_queue.cc:237] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } }
I20260201 03:09:25.041556 7826 catalog_manager.cc:5654] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c68406c7093460f99338c3e563efd4d (127.28.181.2). New cstate: current_term: 1 leader_uuid: "4c68406c7093460f99338c3e563efd4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c68406c7093460f99338c3e563efd4d" member_type: VOTER last_known_addr { host: "127.28.181.2" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260201 03:09:25.046383 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.044684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.28.181.2:36055)
0201 03:09:25.044738 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0201 03:09:25.044740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:09:25.044746 (+ 6us) server_negotiation.cc:408] Connection header received
0201 03:09:25.044785 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:09:25.044785 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:09:25.044797 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:09:25.044846 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:09:25.045081 (+ 235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.045532 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:09:25.046035 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:09:25.046138 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:09:25.046163 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:09:25.046227 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:09:25.046279 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0201 03:09:25.046332 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:09:25.046348 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:09:25.047600 8246 raft_consensus.cc:1275] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 1 FOLLOWER]: Refusing update from remote peer 4c68406c7093460f99338c3e563efd4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:25.047600 7974 raft_consensus.cc:1275] T c9068648263b491b8aff8f36bceca218 P 75f3111db4364119a05b8d79e606c03c [term 1 FOLLOWER]: Refusing update from remote peer 4c68406c7093460f99338c3e563efd4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:09:25.047859 8312 consensus_queue.cc:1048] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af75e546246b4ee0b7e324b99e308142" member_type: VOTER last_known_addr { host: "127.28.181.3" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:25.047950 8316 consensus_queue.cc:1048] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f3111db4364119a05b8d79e606c03c" member_type: VOTER last_known_addr { host: "127.28.181.1" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:09:25.050328 8321 mvcc.cc:204] Tried to move back new op lower bound from 7249573335233007616 to 7249573335208148992. Current Snapshot: MvccSnapshot[applied={T|T < 7249573335233007616}]
I20260201 03:09:25.050509 8320 mvcc.cc:204] Tried to move back new op lower bound from 7249573335233007616 to 7249573335208148992. Current Snapshot: MvccSnapshot[applied={T|T < 7249573335233007616}]
I20260201 03:09:25.051582 7827 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:25.051659 7827 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:09:25.052289 7827 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2709a2e1b90049bcb595ed81fb3695f3: Sending DeleteTablet for 3 replicas of tablet c9068648263b491b8aff8f36bceca218
I20260201 03:09:25.052533 8090 tablet_service.cc:1552] Processing DeleteTablet for tablet c9068648263b491b8aff8f36bceca218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:25 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:35664
I20260201 03:09:25.052600 8226 tablet_service.cc:1552] Processing DeleteTablet for tablet c9068648263b491b8aff8f36bceca218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:25 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:39656
I20260201 03:09:25.052711 7954 tablet_service.cc:1552] Processing DeleteTablet for tablet c9068648263b491b8aff8f36bceca218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:09:25 UTC) from {username='kudu', principal='kudu/127.28.181.62@KRBTEST.COM'} at 127.0.0.1:50364
I20260201 03:09:25.052831 8330 tablet_replica.cc:333] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: stopping tablet replica
I20260201 03:09:25.052877 8329 tablet_replica.cc:333] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d: stopping tablet replica
I20260201 03:09:25.052939 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7891
I20260201 03:09:25.052951 8329 raft_consensus.cc:2243] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:09:25.052960 8330 raft_consensus.cc:2243] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:09:25.053125 8330 raft_consensus.cc:2272] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:25.055107 8330 ts_tablet_manager.cc:1916] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:09:25.056751 8330 ts_tablet_manager.cc:1929] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:09:25.056811 8330 log.cc:1199] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Deleting WAL directory at /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769915279059265-29396-0/minicluster-data/ts-2/wal/wals/c9068648263b491b8aff8f36bceca218
I20260201 03:09:25.057054 8330 ts_tablet_manager.cc:1950] T c9068648263b491b8aff8f36bceca218 P af75e546246b4ee0b7e324b99e308142: Deleting consensus metadata
I20260201 03:09:25.057354 7814 catalog_manager.cc:4985] TS af75e546246b4ee0b7e324b99e308142 (127.28.181.3:36295): tablet c9068648263b491b8aff8f36bceca218 (table test-table [id=791b49cc4e97462eb3f842d6a7b584bd]) successfully deleted
W20260201 03:09:25.058295 7814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.181.1:43257 (error 108)
I20260201 03:09:25.058460 8329 raft_consensus.cc:2272] T c9068648263b491b8aff8f36bceca218 P 4c68406c7093460f99338c3e563efd4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:09:25.058643 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 8027
I20260201 03:09:25.060014 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.059906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.1:43257 (local address 127.0.0.1:50380)
0201 03:09:25.059962 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:25.059991 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.1:43257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260201 03:09:25.060137 7814 catalog_manager.cc:4712] TS 75f3111db4364119a05b8d79e606c03c (127.28.181.1:43257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9068648263b491b8aff8f36bceca218: Network error: Client connection negotiation failed: client connection to 127.28.181.1:43257: connect: Connection refused (error 111)
I20260201 03:09:25.062908 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 8163
I20260201 03:09:25.062913 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:09:25.062826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.2:36055 (local address 127.0.0.1:35692)
0201 03:09:25.062869 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0201 03:09:25.062894 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.2:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260201 03:09:25.067909 29396 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu with pid 7795
2026-02-01T03:09:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2501 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:09:25 dist-test-slave-ck1n krb5kdc[8335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:09:25 dist-test-slave-ck1n krb5kdc[8335](info): set up 2 sockets
Feb 01 03:09:25 dist-test-slave-ck1n krb5kdc[8335](info): commencing operation
krb5kdc: starting...
W20260201 03:09:26.774276 29396 mini_kdc.cc:121] Time spent starting KDC: real 1.687s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/test-user.keytab.
Feb 01 03:09:26 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:09:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:09:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangeradmin_127.28.181.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangeradmin_127.28.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangerlookup_127.28.181.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangerlookup_127.28.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/HTTP_127.28.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/HTTP_127.28.181.20@KRBTEST.COM.keytab.
I20260201 03:09:26.895370 29396 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFbl1Rs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/postgres -l logfile start
2026-02-01 03:09:28.973 UTC [8366] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-01 03:09:28.973 UTC [8366] LOG: listening on IPv4 address "127.28.181.20", port 59401
2026-02-01 03:09:28.976 UTC [8366] LOG: listening on Unix socket "/tmp/.s.PGSQL.59401"
2026-02-01 03:09:28.981 UTC [8371] LOG: database system was shut down at 2026-02-01 03:09:27 UTC
2026-02-01 03:09:28.986 UTC [8366] LOG: database system is ready to accept connections
I20260201 03:09:30.649971 29396 mini_postgres.cc:96] Postgres bound to 59401
2026-02-01 03:09:30.655 UTC [8377] FATAL: database "slave" does not exist
127.28.181.20:59401 - accepting connections
I20260201 03:09:30.655926 29396 mini_ranger.cc:161] Starting Ranger...
I20260201 03:09:30.670717 29396 mini_ranger.cc:84] Created miniranger Postgres user
I20260201 03:09:30.727862 29396 mini_ranger.cc:87] Created ranger Postgres database
I20260201 03:09:30.727943 29396 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin
2026-02-01 03:09:30,966 [I] DB FLAVOR :POSTGRES
2026-02-01 03:09:30,966 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:09:30,966 [I] Checking connection..
2026-02-01 03:09:30,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-01 03:09:31,224 [I] Checking connection passed.
2026-02-01 03:09:31,225 [I] --------- Verifying version history table ---------
2026-02-01 03:09:31,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:09:31,491 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:09:31,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:09:31,734 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:09:31,734 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-01 03:09:31,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-01 03:09:31.983 UTC [8444] WARNING: there is no transaction in progress
2026-02-01 03:09:31,989 [I] create_dbversion_catalog.sql file imported successfully
2026-02-01 03:09:31,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:09:32,257 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-01 03:09:32,257 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:09:32,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-01 03:09:32,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-01 03:09:32,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ck1n.c.gcp-upstream.internal','N') ;"
2026-02-01 03:09:33,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:09:33,286 [I] Table x_portal_user does not exist in database ranger
2026-02-01 03:09:33,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:09:33,538 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-01 03:09:33,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:09:33,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:09:34,026 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-01 03:09:34,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-01 03:09:34.702 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.711 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.719 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.728 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.737 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.749 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.798 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.802 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.810 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.817 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.824 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.832 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.839 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.842 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.845 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:34.848 UTC [8561] WARNING: there is no transaction in progress
2026-02-01 03:09:35,169 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-01 03:09:35,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-01 03:09:35,410 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-01 03:09:35,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:09:35,650 [I] Table x_portal_user already exists in database 'ranger'
2026-02-01 03:09:35,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:09:35,908 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-01 03:09:35,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:09:36,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:09:36,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ck1n.c.gcp-upstream.internal';"
2026-02-01 03:09:36,651 [I] CORE_DB_SCHEMA import status has been updated
2026-02-01 03:09:36,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-01 03:09:36,890 [I] DB_PATCHES have already been applied
I20260201 03:09:36.894750 29396 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFbl1Rs/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260201 03:09:36.894805 29396 mini_ranger.cc:193] Using host: 127.28.181.20
I20260201 03:09:36.897111 29396 mini_ranger.cc:227] Ranger admin URL: http://127.28.181.20:44965
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin/ews/webapp
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-admin/ews/webapp].
Feb 01, 2026 3:09:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260201 03:09:38.824455 29396 mini_ranger.cc:160] Time spent starting Ranger: real 8.169s user 0.000s sys 0.007s
Feb 01, 2026 3:09:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 01, 2026 3:09:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
Feb 01, 2026 3:09:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 01, 2026 3:09:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 01, 2026 3:09:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260201 03:09:55.333105 29396 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangerkms_127.28.181.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/rangerkms_127.28.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.181.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.181.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/HTTP_127.28.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.181.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/HTTP_127.28.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 01 03:09:55 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915395, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260201 03:09:55.398777 29396 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260201 03:09:55.407323 29396 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260201 03:09:55.501663 29396 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260201 03:09:55.501762 29396 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms
I20260201 03:09:55.501804 29396 mini_ranger_kms.cc:227] Using postgres at 127.28.181.20:59401
2026-02-01 03:09:55,647 [I] DB FLAVOR :POSTGRES
2026-02-01 03:09:55,647 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:09:55,647 [I] Checking connection
2026-02-01 03:09:55,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-01 03:09:55,915 [I] connection success
2026-02-01 03:09:55,915 [I] --------- Verifying Ranger DB tables ---------
2026-02-01 03:09:55,915 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-01 03:09:55,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-01 03:09:56,160 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-01 03:09:56,160 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:09:56,160 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-01 03:09:56,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.181.20:59401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-01 03:09:56,427 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260201 03:09:56.817636 29396 mini_ranger_kms.cc:310] Created kms service
I20260201 03:09:57.015506 29396 mini_ranger_kms.cc:326] Created kudu user
I20260201 03:09:57.052168 29396 mini_ranger_kms.cc:343] Created rangerkms user
Feb 01 03:09:57 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915395, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
I20260201 03:09:57.389310 29396 mini_ranger_kms.cc:384] Added ranger policy
I20260201 03:09:57.389467 29396 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFbl1Rs/build/release/bin/postgresql.jar:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFbl1Rs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260201 03:09:57.389488 29396 mini_ranger_kms.cc:240] Using host: 127.28.181.20
I20260201 03:09:57.391790 29396 mini_ranger_kms.cc:276] Ranger KMS PID: 8753
I20260201 03:09:57.391880 29396 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.181.20:38105
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-kms/ews/webapp].
Feb 01, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260201 03:09:59.330273 29396 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 3.931s user 0.000s sys 0.130s
I20260201 03:09:59.330385 29396 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260201 03:09:59.330444 29396 mini_ranger_kms.cc:401] 127.28.181.20:38105/kms/v1/keys
Feb 01 03:09:59 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769915399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.181.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 01 03:09:59 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.181.20@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
Feb 01 03:10:00 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.181.20@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
Feb 01 03:10:01 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:10:01.930603 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:34805
--webserver_interface=127.28.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.181.62:34805
--ranger_config_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.181.20:38105/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:10:01.995625 8806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:10:01.995800 8806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:10:01.995829 8806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:10:01.997005 8806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:10:01.997064 8806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:10:01.997084 8806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:10:01.997102 8806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:10:01.997128 8806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:10:01.998463 8806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.181.20:38105/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.181.62:34805
--ranger_config_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.181.62:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.181.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:10:01.998663 8806 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:10:01.998831 8806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:10:02.000763 8812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:10:02.000867 8806 server_base.cc:1060] running on GCE node
W20260201 03:10:02.000751 8814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:10:02.000767 8811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:10:02.001187 8806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:10:02.001405 8806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:10:02.002542 8806 hybrid_clock.cc:648] HybridClock initialized: now 1769915402002529 us; error 30 us; skew 500 ppm
Feb 01 03:10:01 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:10:02.004508 8806 init.cc:377] Logged in from keytab as kudu/127.28.181.62@KRBTEST.COM (short username kudu)
I20260201 03:10:02.004895 8806 webserver.cc:492] Webserver started at http://127.28.181.62:42275/ using document root <none> and password file <none>
I20260201 03:10:02.005016 8806 fs_manager.cc:362] Metadata directory not provided
I20260201 03:10:02.005043 8806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:10:02.005105 8806 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:10:02 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.62@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
I20260201 03:10:02.048333 8806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data/instance:
uuid: "77d646e464ee41109d711e8bdfcb2040"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "8f4cea9ee8e928223d58b3cbcb08818a"
server_key_iv: "eb39d13a63c3d37702f20739d94414bd"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.048686 8806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal/instance:
uuid: "77d646e464ee41109d711e8bdfcb2040"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "8f4cea9ee8e928223d58b3cbcb08818a"
server_key_iv: "eb39d13a63c3d37702f20739d94414bd"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.049952 8806 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260201 03:10:02.063056 8821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:02.063337 8806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260201 03:10:02.063474 8806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal
uuid: "77d646e464ee41109d711e8bdfcb2040"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "8f4cea9ee8e928223d58b3cbcb08818a"
server_key_iv: "eb39d13a63c3d37702f20739d94414bd"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.063582 8806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:10:02.074693 8806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:10:02.075892 8806 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:10:02.075975 8806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:10:02.079401 8806 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.62:34805
I20260201 03:10:02.079453 8873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.62:34805 every 8 connection(s)
I20260201 03:10:02.079710 8806 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/data/info.pb
I20260201 03:10:02.080215 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:10:02.082003 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040: Bootstrap starting.
I20260201 03:10:02.082473 8874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040: Neither blocks nor log segments found. Creating new log.
I20260201 03:10:02.082695 8874 log.cc:826] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040: Log is configured to *not* fsync() on all Append() calls
I20260201 03:10:02.083300 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040: No bootstrap required, opened a new log
I20260201 03:10:02.084311 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 8806
I20260201 03:10:02.084414 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/wal/instance
I20260201 03:10:02.084442 8874 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } }
I20260201 03:10:02.084564 8874 raft_consensus.cc:385] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:10:02.084579 29396 external_mini_cluster.cc:1468] Setting key a566c0b4c2c30208177299e1e122aba0
I20260201 03:10:02.084589 8874 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d646e464ee41109d711e8bdfcb2040, State: Initialized, Role: FOLLOWER
I20260201 03:10:02.084689 8874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } }
I20260201 03:10:02.084756 8874 raft_consensus.cc:399] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:10:02.084805 8874 raft_consensus.cc:493] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:10:02.084842 8874 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:10:02.085330 8874 raft_consensus.cc:515] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } }
I20260201 03:10:02.085438 8874 leader_election.cc:304] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77d646e464ee41109d711e8bdfcb2040; no voters:
I20260201 03:10:02.085585 8874 leader_election.cc:290] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:10:02.085613 8879 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:10:02.085703 8879 raft_consensus.cc:697] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [term 1 LEADER]: Becoming Leader. State: Replica: 77d646e464ee41109d711e8bdfcb2040, State: Running, Role: LEADER
I20260201 03:10:02.085803 8874 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:10:02.085783 8879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } }
I20260201 03:10:02.086032 8879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77d646e464ee41109d711e8bdfcb2040. Latest consensus state: current_term: 1 leader_uuid: "77d646e464ee41109d711e8bdfcb2040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } } }
I20260201 03:10:02.086171 8879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [sys.catalog]: This master's current role is: LEADER
I20260201 03:10:02.086242 8881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77d646e464ee41109d711e8bdfcb2040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77d646e464ee41109d711e8bdfcb2040" member_type: VOTER last_known_addr { host: "127.28.181.62" port: 34805 } } }
I20260201 03:10:02.086308 8881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040 [sys.catalog]: This master's current role is: LEADER
I20260201 03:10:02.086408 8883 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:10:02.086781 8883 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:10:02.087520 8874 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Feb 01 03:10:02 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:10:02.091935 8896 mvcc.cc:204] Tried to move back new op lower bound from 7249573486965854208 to 7249573486943694848. Current Snapshot: MvccSnapshot[applied={T|T < 7249573486965854208}]
I20260201 03:10:02.095263 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:02.084966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.28.181.62:34805)
0201 03:10:02.085099 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:10:02.085109 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:10:02.085145 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:10:02.085229 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:10:02.085233 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:10:02.085273 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:10:02.085369 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:10:02.086200 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.087199 (+ 999us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:10:02.087849 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.088059 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:10:02.090391 (+ 2332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:10:02.090549 (+ 158us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:10:02.090550 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:10:02.090662 (+ 112us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:10:02.092985 (+ 2323us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:10:02.093340 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:10:02.093342 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:10:02.093342 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:10:02.093396 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:10:02.093568 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:10:02.093569 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:10:02.093569 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:10:02.093832 (+ 263us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:10:02.094905 (+ 1073us) server_negotiation.cc:1036] Waiting for connection context
0201 03:10:02.095008 (+ 103us) server_negotiation.cc:300] Negotiation successful
0201 03:10:02.095111 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260201 03:10:02.097873 8883 catalog_manager.cc:1357] Generated new cluster ID: 15deda9c51664e4880a66eca10de0cad
I20260201 03:10:02.097921 8883 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:10:02.118160 8883 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:10:02.118765 8883 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:10:02.128774 8883 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 77d646e464ee41109d711e8bdfcb2040: Generated new TSK 0
I20260201 03:10:02.128916 8883 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
I20260201 03:10:02.713760 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.181.20:38105/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.1:0
--local_ip_for_outbound_sockets=127.28.181.1
--webserver_interface=127.28.181.1
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34805
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:10:02.794592 8920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:10:02.794740 8920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:10:02.794759 8920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:10:02.795966 8920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:10:02.796005 8920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:10:02.796025 8920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.1
I20260201 03:10:02.797382 8920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.181.20:38105/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.181.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.1
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:10:02.797603 8920 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:10:02.797762 8920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:10:02.799947 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:10:02.799975 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:10:02.800067 8920 server_base.cc:1060] running on GCE node
W20260201 03:10:02.800163 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:10:02.800338 8920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:10:02.800536 8920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:10:02.801681 8920 hybrid_clock.cc:648] HybridClock initialized: now 1769915402801660 us; error 31 us; skew 500 ppm
Feb 01 03:10:02 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:10:02.804242 8920 init.cc:377] Logged in from keytab as kudu/127.28.181.1@KRBTEST.COM (short username kudu)
I20260201 03:10:02.804669 8920 webserver.cc:492] Webserver started at http://127.28.181.1:35293/ using document root <none> and password file <none>
I20260201 03:10:02.804841 8920 fs_manager.cc:362] Metadata directory not provided
I20260201 03:10:02.804888 8920 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:10:02.804992 8920 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:10:02 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
I20260201 03:10:02.816836 8920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data/instance:
uuid: "b5dcc4b7bb1e492c84aaa9ed8479704e"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "54ddcaee7280b06368c07a8cf308a12c"
server_key_iv: "8f46e65fd861fa0558a1237f7e06cea5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.817225 8920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance:
uuid: "b5dcc4b7bb1e492c84aaa9ed8479704e"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "54ddcaee7280b06368c07a8cf308a12c"
server_key_iv: "8f46e65fd861fa0558a1237f7e06cea5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.819233 8920 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260201 03:10:02.835415 8937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:02.845875 8920 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.000s sys 0.001s
I20260201 03:10:02.845981 8920 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal
uuid: "b5dcc4b7bb1e492c84aaa9ed8479704e"
format_stamp: "Formatted at 2026-02-01 03:10:02 on dist-test-slave-ck1n"
server_key: "54ddcaee7280b06368c07a8cf308a12c"
server_key_iv: "8f46e65fd861fa0558a1237f7e06cea5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:02.846060 8920 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:10:02.859684 8920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:10:02.861526 8920 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:10:02.861689 8920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:10:02.861913 8920 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:10:02.862262 8920 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:10:02.862335 8920 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:02.862423 8920 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:10:02.862493 8920 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:02.868785 8920 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.1:36429
I20260201 03:10:02.868849 9050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.1:36429 every 8 connection(s)
I20260201 03:10:02.869155 8920 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:10:02 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.1@KRBTEST.COM for kudu/127.28.181.62@KRBTEST.COM
I20260201 03:10:02.877941 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 8920
I20260201 03:10:02.878041 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-0/wal/instance
I20260201 03:10:02.878082 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:02.869709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:39023)
0201 03:10:02.870121 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:02.870147 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:10:02.871088 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:10:02.871330 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:10:02.871334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:10:02.871553 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:10:02.871912 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:10:02.871916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.872748 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:10:02.872749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:10:02.873284 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:10:02.873286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.873414 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:10:02.873889 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:10:02.873910 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:10:02.875337 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:10:02.876946 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:10:02.876952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:10:02.876954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:10:02.877167 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:10:02.877318 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:10:02.877319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:10:02.877319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:10:02.877392 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:10:02.877605 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:10:02.877609 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:10:02.877726 (+ 117us) client_negotiation.cc:769] Sending connection context
0201 03:10:02.877821 (+ 95us) client_negotiation.cc:240] Negotiation successful
0201 03:10:02.877915 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
I20260201 03:10:02.878193 29396 external_mini_cluster.cc:1468] Setting key 7ef7e0c458aa9a4942ea50a6d9228b06
I20260201 03:10:02.878829 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:02.869880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.181.1:39023 (local address 127.28.181.62:34805)
0201 03:10:02.870036 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0201 03:10:02.870038 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0201 03:10:02.870231 (+ 193us) server_negotiation.cc:408] Connection header received
0201 03:10:02.871192 (+ 961us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:10:02.871193 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:10:02.871205 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:10:02.871277 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:10:02.872011 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.872665 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:10:02.873373 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:10:02.873862 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:10:02.875430 (+ 1568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:10:02.875434 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:10:02.875434 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:10:02.875440 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:10:02.876862 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:10:02.877241 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:10:02.877242 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:10:02.877243 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:10:02.877263 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:10:02.877448 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:10:02.877448 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:10:02.877449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:10:02.877536 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:10:02.878633 (+ 1097us) server_negotiation.cc:1036] Waiting for connection context
0201 03:10:02.878670 (+ 37us) server_negotiation.cc:300] Negotiation successful
0201 03:10:02.878750 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260201 03:10:02.879257 9051 heartbeater.cc:344] Connected to a master server at 127.28.181.62:34805
I20260201 03:10:02.879328 9051 heartbeater.cc:461] Registering TS with master...
I20260201 03:10:02.879494 9051 heartbeater.cc:507] Master 127.28.181.62:34805 requested a full tablet report, sending...
I20260201 03:10:02.879911 8838 ts_manager.cc:194] Registered new tserver with Master: b5dcc4b7bb1e492c84aaa9ed8479704e (127.28.181.1:36429)
I20260201 03:10:02.880713 8838 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.181.1@KRBTEST.COM'} at 127.28.181.1:39023
WARNING: no policy specified for kudu/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.181.2@KRBTEST.COM" created.
Exception in thread "main" Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab.
org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.28.181.62 from keytab /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)I20260201 03:10:02.924289 29396 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
/tmp/dist-test-taskFbl1Rs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.181.20:38105/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.181.2:0
--local_ip_for_outbound_sockets=127.28.181.2
--webserver_interface=127.28.181.2
--webserver_port=0
--tserver_master_addrs=127.28.181.62:34805
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.181.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260201 03:10:02.931187 8912 server.cc:273] Received an EOF from the subprocess
F20260201 03:10:02.931957 8899 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f32eef7cdcd google::LogMessage::Fail() at ??:0
@ 0x7f32eef80b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f32eef7c7cc google::LogMessage::Flush() at ??:0
@ 0x7f32eef7df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f32f0388303 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f32f0074907 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f32f03a96db start_thread at ??:0
@ 0x7f32ee4d171f clone at ??:0
W20260201 03:10:03.012895 9061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:10:03.013031 9061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:10:03.013067 9061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:10:03.014385 9061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:10:03.014418 9061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:10:03.014439 9061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.181.2
I20260201 03:10:03.015743 9061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.181.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.181.20:38105/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.181.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.181.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.181.2
--log_dir=/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 01 Feb 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10287
I20260201 03:10:03.015904 9061 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:10:03.016044 9061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:10:03.018150 9061 server_base.cc:1060] running on GCE node
W20260201 03:10:03.018105 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:10:03.018092 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:10:03.018368 9078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:10:03.018605 9061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:10:03.018797 9061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:10:03.019934 9061 hybrid_clock.cc:648] HybridClock initialized: now 1769915403019924 us; error 27 us; skew 500 ppm
Feb 01 03:10:03 dist-test-slave-ck1n krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:10:03.021842 9061 init.cc:377] Logged in from keytab as kudu/127.28.181.2@KRBTEST.COM (short username kudu)
I20260201 03:10:03.022286 9061 webserver.cc:492] Webserver started at http://127.28.181.2:33273/ using document root <none> and password file <none>
I20260201 03:10:03.022452 9061 fs_manager.cc:362] Metadata directory not provided
I20260201 03:10:03.022500 9061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:10:03.022605 9061 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:10:03 dist-test-slave-ck1n krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.181.2@KRBTEST.COM for HTTP/127.28.181.20@KRBTEST.COM
I20260201 03:10:03.033326 9061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data/instance:
uuid: "67091a59210641b68d9108ce13b19bf7"
format_stamp: "Formatted at 2026-02-01 03:10:03 on dist-test-slave-ck1n"
server_key: "6ebff717f90e54a6b94c9def51f48fbe"
server_key_iv: "4d697177a659fe58436181b0c4a73eb5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:03.033608 9061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance:
uuid: "67091a59210641b68d9108ce13b19bf7"
format_stamp: "Formatted at 2026-02-01 03:10:03 on dist-test-slave-ck1n"
server_key: "6ebff717f90e54a6b94c9def51f48fbe"
server_key_iv: "4d697177a659fe58436181b0c4a73eb5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:03.034674 9061 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:10:03.050135 9085 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:03.050326 9061 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260201 03:10:03.050401 9061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal
uuid: "67091a59210641b68d9108ce13b19bf7"
format_stamp: "Formatted at 2026-02-01 03:10:03 on dist-test-slave-ck1n"
server_key: "6ebff717f90e54a6b94c9def51f48fbe"
server_key_iv: "4d697177a659fe58436181b0c4a73eb5"
server_key_version: "kuduclusterkey@0"
I20260201 03:10:03.050468 9061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:10:03.076869 9061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:10:03.078526 9061 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:10:03.078658 9061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:10:03.078861 9061 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:10:03.079154 9061 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:10:03.079200 9061 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:03.079252 9061 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:10:03.079278 9061 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:10:03.085354 9061 rpc_server.cc:307] RPC server started. Bound to: 127.28.181.2:38133
I20260201 03:10:03.085652 9061 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/data/info.pb
I20260201 03:10:03.085899 9198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.181.2:38133 every 8 connection(s)
I20260201 03:10:03.088450 29396 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskFbl1Rs/build/release/bin/kudu as pid 9061
I20260201 03:10:03.088550 29396 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskFbl1Rs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769915279059265-29396-0/minicluster-data/ts-1/wal/instance
I20260201 03:10:03.088725 29396 external_mini_cluster.cc:1468] Setting key 4495dd3dd3247e8c9366b7c57bdea594
W20260201 03:10:03.124071 8333 connection.cc:537] client connection to 127.28.181.62:34805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260201 03:10:03.124455 9201 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:10:03.086485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:46327)
0201 03:10:03.086607 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:03.086633 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0201 03:10:03.087339 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:10:03.124317 (+ 36978us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
W20260201 03:10:03.124709 9199 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:34805 (0 consecutive failures): Network error: Failed to ping master at 127.28.181.62:34805: Client connection negotiation failed: client connection to 127.28.181.62:34805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260201 03:10:03.124994 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:03.124899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:50723)
0201 03:10:03.124945 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:03.124976 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260201 03:10:03.125298 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:03.125225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:59839)
0201 03:10:03.125265 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:03.125285 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260201 03:10:03.125408 9199 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260201 03:10:03.881577 9051 heartbeater.cc:499] Master 127.28.181.62:34805 was elected leader, sending a full tablet report...
I20260201 03:10:03.882112 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:03.881884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:34531)
0201 03:10:03.882018 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:03.882068 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":55,"threads_started":1}
W20260201 03:10:03.882306 9051 heartbeater.cc:646] Failed to heartbeat to 127.28.181.62:34805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
I20260201 03:10:04.126073 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:04.125870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:34279)
0201 03:10:04.125987 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:04.126020 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260201 03:10:04.882962 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:04.882786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:36657)
0201 03:10:04.882898 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:04.882931 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260201 03:10:05.126874 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:05.126649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:52221)
0201 03:10:05.126784 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:05.126816 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260201 03:10:05.883668 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:05.883481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:49889)
0201 03:10:05.883601 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:05.883635 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260201 03:10:06.127437 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:06.127269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:42977)
0201 03:10:06.127368 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:06.127398 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":64,"threads_started":1}
I20260201 03:10:06.884400 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:06.884180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:39819)
0201 03:10:06.884315 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:06.884356 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260201 03:10:07.128216 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:07.128017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:48281)
0201 03:10:07.128136 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:07.128174 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260201 03:10:07.885161 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:07.884964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.1:39709)
0201 03:10:07.885083 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:07.885119 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
F20260201 03:10:08.121914 29396 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1769915408 (unix time) try "date -d @1769915408" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000072d4) received by PID 29396 (TID 0x7f4a08979540) from PID 29396; stack trace: ***
@ 0x7f4a1201a980 (unknown) at ??:0
@ 0x7f4a1026dfb7 gsignal at ??:0
I20260201 03:10:08.128849 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:10:08.128648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.181.62:34805 (local address 127.28.181.2:56065)
0201 03:10:08.128760 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0201 03:10:08.128792 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.181.62:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":49,"threads_started":1}
@ 0x7f4a1026f921 abort at ??:0
@ 0x7f4a110fedcd google::LogMessage::Fail() at ??:0
@ 0x7f4a11102b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4a110fe7cc google::LogMessage::Flush() at ??:0
@ 0x7f4a110fff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a128f9d38 kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f4a128fb6d1 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5607dc64ccc6 kudu::SecurityITest::StartCluster() at ??:0
@ 0x5607dc624c74 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f4a11c300ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f4a11c24bdb testing::Test::Run() at ??:0
@ 0x7f4a11c24d9d testing::TestInfo::Run() at ??:0
@ 0x7f4a11c25377 testing::TestSuite::Run() at ??:0
@ 0x7f4a11c2577c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f4a11c3060d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f4a11c24e63 testing::UnitTest::Run() at ??:0
@ 0x5607dc617807 main at ??:0
@ 0x7f4a10250bf7 __libc_start_main at ??:0
@ 0x5607dc617c5a _start at ??:0
2026-02-01 03:10:08.276 UTC [8784] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:10:08.276 UTC [8680] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:10:08.276 UTC [8678] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:10:08.277 UTC [8676] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:10:08.278 UTC [8677] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:10:08.278 UTC [8681] FATAL: terminating connection due to unexpected postmaster exit