[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:24 dist-test-slave-0bdw krb5kdc[10431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:24 dist-test-slave-0bdw krb5kdc[10431](info): set up 2 sockets
Apr 30 20:01:24 dist-test-slave-0bdw krb5kdc[10431](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260430 20:01:26.421525 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.059s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:26 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:26.605612 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41257
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:41257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:26.749890 10447 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:26.750306 10447 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:26.750377 10447 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:26.755458 10447 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:26.755569 10447 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:26.755597 10447 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:26.755620 10447 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:26.755640 10447 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:26.762490 10447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:41257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:26.764079 10447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:26.765591 10447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 20:01:26.774374 10447 server_base.cc:1061] running on GCE node
W20260430 20:01:26.774439 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:26.774663 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:26.774439 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:26.775429 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:26.776669 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:26.777894 10447 hybrid_clock.cc:648] HybridClock initialized: now 1777579286777844 us; error 79 us; skew 500 ppm
Apr 30 20:01:26 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:26.781896 10447 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:26.783219 10447 webserver.cc:492] Webserver started at http://127.10.46.62:44243/ using document root <none> and password file <none>
I20260430 20:01:26.783941 10447 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:26.784000 10447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:26.784209 10447 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:26.786072 10447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "47d97c616949413fb43a17ed0bbb44eb"
format_stamp: "Formatted at 2026-04-30 20:01:26 on dist-test-slave-0bdw"
server_key: "f7e7e985d85e0e7250a1b08617a4cab1"
server_key_iv: "ffe6ecfa67c1b962be3eb71028b87e10"
server_key_version: "encryptionkey@0"
I20260430 20:01:26.786571 10447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "47d97c616949413fb43a17ed0bbb44eb"
format_stamp: "Formatted at 2026-04-30 20:01:26 on dist-test-slave-0bdw"
server_key: "f7e7e985d85e0e7250a1b08617a4cab1"
server_key_iv: "ffe6ecfa67c1b962be3eb71028b87e10"
server_key_version: "encryptionkey@0"
I20260430 20:01:26.790861 10447 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:01:26.793936 10462 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:26.795368 10447 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:26.795575 10447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "47d97c616949413fb43a17ed0bbb44eb"
format_stamp: "Formatted at 2026-04-30 20:01:26 on dist-test-slave-0bdw"
server_key: "f7e7e985d85e0e7250a1b08617a4cab1"
server_key_iv: "ffe6ecfa67c1b962be3eb71028b87e10"
server_key_version: "encryptionkey@0"
I20260430 20:01:26.795717 10447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:26.812177 10447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:26.815834 10447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:26.816246 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:26.825803 10447 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:41257
I20260430 20:01:26.825798 10514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:41257 every 8 connection(s)
I20260430 20:01:26.827142 10447 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:26.831337 10515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:26.835311 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 10447
I20260430 20:01:26.835530 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:26.837419 10424 external_mini_cluster.cc:1468] Setting key ddcdc3aff27424587a8b9aac3d8ee09b
I20260430 20:01:26.840075 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: Bootstrap starting.
I20260430 20:01:26.842628 10515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:26.843499 10515 log.cc:826] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:26.846285 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: No bootstrap required, opened a new log
Apr 30 20:01:26 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:26.852253 10515 raft_consensus.cc:359] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } }
I20260430 20:01:26.852741 10515 raft_consensus.cc:385] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:26.852975 10515 raft_consensus.cc:740] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d97c616949413fb43a17ed0bbb44eb, State: Initialized, Role: FOLLOWER
I20260430 20:01:26.853883 10515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } }
I20260430 20:01:26.854887 10515 raft_consensus.cc:399] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:26.855436 10515 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:26.856195 10515 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:26.857645 10515 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } }
I20260430 20:01:26.858143 10515 leader_election.cc:304] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d97c616949413fb43a17ed0bbb44eb; no voters:
I20260430 20:01:26.858549 10515 leader_election.cc:290] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:26.858973 10520 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:26.859416 10520 raft_consensus.cc:697] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [term 1 LEADER]: Becoming Leader. State: Replica: 47d97c616949413fb43a17ed0bbb44eb, State: Running, Role: LEADER
I20260430 20:01:26.859862 10520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } }
I20260430 20:01:26.860054 10515 sys_catalog.cc:565] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:26.861938 10518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:26.840474 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.10.46.62:41257)
0430 20:01:26.841034 (+ 560us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:26.841045 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:26.841417 (+ 372us) server_negotiation.cc:408] Connection header received
0430 20:01:26.844075 (+ 2658us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:26.844095 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:26.844422 (+ 327us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:26.844720 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:26.846847 (+ 2127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:26.847616 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:26.848403 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:26.848685 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:26.853399 (+ 4714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:26.853430 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:26.853443 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:26.853612 (+ 169us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:26.857717 (+ 4105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:26.858499 (+ 782us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:26.858509 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:26.858515 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:26.858628 (+ 113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:26.859031 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:26.859035 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:26.859037 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:26.859484 (+ 447us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:26.859688 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:26.860811 (+ 1123us) server_negotiation.cc:300] Negotiation successful
0430 20:01:26.861067 (+ 256us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":144,"threads_started":1}
I20260430 20:01:26.862182 10521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d97c616949413fb43a17ed0bbb44eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } } }
I20260430 20:01:26.862433 10522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d97c616949413fb43a17ed0bbb44eb. Latest consensus state: current_term: 1 leader_uuid: "47d97c616949413fb43a17ed0bbb44eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d97c616949413fb43a17ed0bbb44eb" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41257 } } }
I20260430 20:01:26.862704 10521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:26.862951 10522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb [sys.catalog]: This master's current role is: LEADER
W20260430 20:01:26.868269 10534 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:01:26.868410 10534 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:01:26.868640 10535 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:26.870079 10535 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:26.879328 10535 catalog_manager.cc:1357] Generated new cluster ID: 86c799bc7dc6407288cd93f5faac5013
I20260430 20:01:26.879547 10535 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:26.901386 10535 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:26.902499 10535 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:26.911826 10535 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 47d97c616949413fb43a17ed0bbb44eb: Generated new TSK 0
I20260430 20:01:26.913097 10535 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:26.980590 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41257
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:27.125653 10543 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:27.125928 10543 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:27.126030 10543 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:27.129873 10543 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:27.129974 10543 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:27.130062 10543 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:27.135099 10543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:27.136498 10543 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:27.137475 10543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:27.145493 10549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:27.145493 10551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:27.145578 10548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:27.145968 10543 server_base.cc:1061] running on GCE node
I20260430 20:01:27.146502 10543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:27.147572 10543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:27.148860 10543 hybrid_clock.cc:648] HybridClock initialized: now 1777579287148822 us; error 58 us; skew 500 ppm
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:27.152419 10543 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:27.154614 10543 webserver.cc:492] Webserver started at http://127.10.46.1:45417/ using document root <none> and password file <none>
I20260430 20:01:27.156273 10543 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:27.156589 10543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:27.157127 10543 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:27.159320 10543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "c6590e17622b474e9110eee6c2417573"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "139c47ff8e256fa144209acdcc69b652"
server_key_iv: "bdb970fd806ac01f9431de8697e69603"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.160159 10543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "c6590e17622b474e9110eee6c2417573"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "139c47ff8e256fa144209acdcc69b652"
server_key_iv: "bdb970fd806ac01f9431de8697e69603"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.165134 10543 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260430 20:01:27.168622 10558 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.171211 10543 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:01:27.171430 10543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "c6590e17622b474e9110eee6c2417573"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "139c47ff8e256fa144209acdcc69b652"
server_key_iv: "bdb970fd806ac01f9431de8697e69603"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.171589 10543 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:27.194584 10543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:27.198894 10543 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:27.199095 10543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:27.199679 10543 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:27.200800 10543 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:27.200860 10543 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.200906 10543 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:27.200922 10543 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.216749 10543 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:41777
I20260430 20:01:27.216879 10671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:41777 every 8 connection(s)
I20260430 20:01:27.217914 10543 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:27.221254 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 10543
I20260430 20:01:27.221382 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:27.221662 10424 external_mini_cluster.cc:1468] Setting key 39b66dd5a40f458b6e0ab0e7e6439c78
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:01:27.235404 10518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.220215 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:55561 (local address 127.10.46.62:41257)
0430 20:01:27.220400 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:27.220404 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:27.221260 (+ 856us) server_negotiation.cc:408] Connection header received
0430 20:01:27.222375 (+ 1115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:27.222379 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:27.222445 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:27.222536 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:27.224902 (+ 2366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.225768 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.227127 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.227397 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.230351 (+ 2954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:27.230391 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:27.230397 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:27.230442 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:27.232313 (+ 1871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.233105 (+ 792us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.233108 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.233110 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.233164 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.233628 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.233633 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.233635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.233909 (+ 274us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:27.234058 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:27.234751 (+ 693us) server_negotiation.cc:300] Negotiation successful
0430 20:01:27.235054 (+ 303us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 20:01:27.235960 10674 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.220497 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41257 (local address 127.10.46.1:55561)
0430 20:01:27.221084 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:27.221128 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:27.222147 (+ 1019us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:27.222745 (+ 598us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:27.222755 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:27.223327 (+ 572us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:27.224639 (+ 1312us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.224661 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.226095 (+ 1434us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.226102 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:27.226809 (+ 707us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.226829 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.227358 (+ 529us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.228193 (+ 835us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:27.228217 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:27.230121 (+ 1904us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:27.232520 (+ 2399us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.232528 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.232540 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.232973 (+ 433us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.233280 (+ 307us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.233283 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.233284 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.233424 (+ 140us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.234122 (+ 698us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:27.234128 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:27.234430 (+ 302us) client_negotiation.cc:770] Sending connection context
0430 20:01:27.234638 (+ 208us) client_negotiation.cc:241] Negotiation successful
0430 20:01:27.235004 (+ 366us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":115,"threads_started":1}
I20260430 20:01:27.238008 10672 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41257
I20260430 20:01:27.238315 10672 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:27.238888 10672 heartbeater.cc:507] Master 127.10.46.62:41257 requested a full tablet report, sending...
I20260430 20:01:27.240847 10479 ts_manager.cc:194] Registered new tserver with Master: c6590e17622b474e9110eee6c2417573 (127.10.46.1:41777)
I20260430 20:01:27.242115 10479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:55561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:27.296800 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41257
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:27.425860 10679 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:27.426462 10679 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:27.426833 10679 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:27.431629 10679 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:27.431795 10679 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:27.431972 10679 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:27.437394 10679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:27.438985 10679 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:27.440109 10679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:27.447938 10685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:27.447938 10687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:27.448421 10684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:27.449357 10679 server_base.cc:1061] running on GCE node
I20260430 20:01:27.450330 10679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:27.451463 10679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:27.452744 10679 hybrid_clock.cc:648] HybridClock initialized: now 1777579287452817 us; error 162 us; skew 500 ppm
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:27.457221 10679 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:27.458670 10679 webserver.cc:492] Webserver started at http://127.10.46.2:45271/ using document root <none> and password file <none>
I20260430 20:01:27.459359 10679 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:27.459445 10679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:27.459677 10679 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:27.461720 10679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "79d651fe967a4efa9eab7470cb84e967"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "442bf294f4ccbe5adb0ae20a8b4fd613"
server_key_iv: "e9b2e55e20df4725fa39c653ea969f33"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.462277 10679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "79d651fe967a4efa9eab7470cb84e967"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "442bf294f4ccbe5adb0ae20a8b4fd613"
server_key_iv: "e9b2e55e20df4725fa39c653ea969f33"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.466514 10679 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:01:27.470212 10694 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.471889 10679 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:27.472074 10679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "79d651fe967a4efa9eab7470cb84e967"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "442bf294f4ccbe5adb0ae20a8b4fd613"
server_key_iv: "e9b2e55e20df4725fa39c653ea969f33"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.472234 10679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:27.486173 10679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:27.490121 10679 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:27.490314 10679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:27.490892 10679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:27.491895 10679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:27.491950 10679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.491993 10679 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:27.492009 10679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.507679 10679 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:45341
I20260430 20:01:27.507707 10807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:45341 every 8 connection(s)
I20260430 20:01:27.508838 10679 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:27.516651 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 10679
I20260430 20:01:27.516824 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:27.517113 10424 external_mini_cluster.cc:1468] Setting key 6e01d8bedee69470f120c820a165fc39
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:27.527812 10518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.511854 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:50923 (local address 127.10.46.62:41257)
0430 20:01:27.512049 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:27.512053 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:27.513058 (+ 1005us) server_negotiation.cc:408] Connection header received
0430 20:01:27.514115 (+ 1057us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:27.514120 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:27.514210 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:27.514354 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:27.516359 (+ 2005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.517251 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.518280 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.518514 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.521751 (+ 3237us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:27.521784 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:27.521788 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:27.521837 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:27.523971 (+ 2134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.524955 (+ 984us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.524963 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.524968 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.525042 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.525710 (+ 668us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.525717 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.525720 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.525960 (+ 240us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:27.526114 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:27.527355 (+ 1241us) server_negotiation.cc:300] Negotiation successful
0430 20:01:27.527554 (+ 199us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:27.528337 10810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.512290 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41257 (local address 127.10.46.2:50923)
0430 20:01:27.512848 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:27.512892 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:27.513805 (+ 913us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:27.514566 (+ 761us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:27.514576 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:27.515016 (+ 440us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:27.516055 (+ 1039us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.516072 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.517526 (+ 1454us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.517530 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:27.518062 (+ 532us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.518072 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.518326 (+ 254us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.519005 (+ 679us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:27.519027 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:27.521443 (+ 2416us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:27.524437 (+ 2994us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.524445 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.524460 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.524792 (+ 332us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.525292 (+ 500us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.525298 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.525301 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.525515 (+ 214us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.526158 (+ 643us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:27.526166 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:27.526570 (+ 404us) client_negotiation.cc:770] Sending connection context
0430 20:01:27.527039 (+ 469us) client_negotiation.cc:241] Negotiation successful
0430 20:01:27.527360 (+ 321us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":147,"threads_started":1}
I20260430 20:01:27.530033 10808 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41257
I20260430 20:01:27.530400 10808 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:27.531032 10808 heartbeater.cc:507] Master 127.10.46.62:41257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
I20260430 20:01:27.532812 10479 ts_manager.cc:194] Registered new tserver with Master: 79d651fe967a4efa9eab7470cb84e967 (127.10.46.2:45341)
I20260430 20:01:27.533460 10479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:50923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:27.585453 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41257
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:27.728072 10815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:27.728394 10815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:27.728471 10815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:27.732913 10815 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:27.733023 10815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:27.733124 10815 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:27.739317 10815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:27.740833 10815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:27.741755 10815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:27.751999 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:27.752028 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:27.752173 10815 server_base.cc:1061] running on GCE node
W20260430 20:01:27.752390 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:27.752887 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:27.753688 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:27.754902 10815 hybrid_clock.cc:648] HybridClock initialized: now 1777579287754882 us; error 38 us; skew 500 ppm
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:27.761408 10815 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:27.763437 10815 webserver.cc:492] Webserver started at http://127.10.46.3:43879/ using document root <none> and password file <none>
I20260430 20:01:27.764160 10815 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:27.764223 10815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:27.764411 10815 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:27.766355 10815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "214ce4385a8744988261929a775a8a03"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "b3434c2d35e2315cf00152f5d0d9dc6e"
server_key_iv: "71b73615762eafa044ee619cdbecfd8d"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.766873 10815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "214ce4385a8744988261929a775a8a03"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "b3434c2d35e2315cf00152f5d0d9dc6e"
server_key_iv: "71b73615762eafa044ee619cdbecfd8d"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.772330 10815 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260430 20:01:27.775400 10830 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.777352 10815 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:27.777555 10815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "214ce4385a8744988261929a775a8a03"
format_stamp: "Formatted at 2026-04-30 20:01:27 on dist-test-slave-0bdw"
server_key: "b3434c2d35e2315cf00152f5d0d9dc6e"
server_key_iv: "71b73615762eafa044ee619cdbecfd8d"
server_key_version: "encryptionkey@0"
I20260430 20:01:27.777693 10815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:27.794255 10815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:27.798679 10815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:27.798877 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:27.799851 10815 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:27.801491 10815 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:27.801615 10815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.801972 10815 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:27.802062 10815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:27.815984 10815 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:44119
I20260430 20:01:27.816017 10943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:44119 every 8 connection(s)
I20260430 20:01:27.817521 10815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:27.819662 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 10815
I20260430 20:01:27.819808 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:27.820143 10424 external_mini_cluster.cc:1468] Setting key 996966071fc81b76da2b78dffaf3f644
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:27.835057 10518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.819967 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:58073 (local address 127.10.46.62:41257)
0430 20:01:27.820280 (+ 313us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:27.820286 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:27.821810 (+ 1524us) server_negotiation.cc:408] Connection header received
0430 20:01:27.823083 (+ 1273us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:27.823088 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:27.823189 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:27.823333 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:27.825736 (+ 2403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.826420 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.827509 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.827692 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.830927 (+ 3235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:27.830954 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:27.830957 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:27.830992 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:27.832755 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.833421 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.833425 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.833426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.833477 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.833827 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.833831 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.833833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.834020 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:27.834139 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:27.834717 (+ 578us) server_negotiation.cc:300] Negotiation successful
0430 20:01:27.834860 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260430 20:01:27.836544 10946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.820467 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41257 (local address 127.10.46.3:58073)
0430 20:01:27.821579 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:27.821649 (+ 70us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:27.822693 (+ 1044us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:27.823700 (+ 1007us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:27.823714 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:27.824402 (+ 688us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:27.825464 (+ 1062us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.825485 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.826628 (+ 1143us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.826632 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:27.827296 (+ 664us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:27.827306 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.827567 (+ 261us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.828323 (+ 756us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:27.828349 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:27.830676 (+ 2327us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:27.832932 (+ 2256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.832941 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.832957 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.833271 (+ 314us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.833597 (+ 326us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:27.833600 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:27.833602 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:27.833710 (+ 108us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:27.834163 (+ 453us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:27.834170 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:27.834431 (+ 261us) client_negotiation.cc:770] Sending connection context
0430 20:01:27.834660 (+ 229us) client_negotiation.cc:241] Negotiation successful
0430 20:01:27.834980 (+ 320us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":463,"threads_started":1}
I20260430 20:01:27.837908 10944 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41257
I20260430 20:01:27.838203 10944 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:27.838837 10944 heartbeater.cc:507] Master 127.10.46.62:41257 requested a full tablet report, sending...
I20260430 20:01:27.840215 10479 ts_manager.cc:194] Registered new tserver with Master: 214ce4385a8744988261929a775a8a03 (127.10.46.3:44119)
I20260430 20:01:27.840793 10479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:58073
I20260430 20:01:27.851594 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:27.886627 10951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.874768 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.10.46.1:41777)
0430 20:01:27.875370 (+ 602us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:27.875387 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:27.875418 (+ 31us) server_negotiation.cc:408] Connection header received
0430 20:01:27.875482 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:27.875487 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:27.875679 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:27.875952 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:27.877082 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.877966 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.878809 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.879295 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.882251 (+ 2956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:27.882289 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:27.882303 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:27.882339 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:27.884680 (+ 2341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.885239 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.885244 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.885246 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.885301 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.885578 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.885581 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.885583 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.885785 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:27.886001 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:27.886232 (+ 231us) server_negotiation.cc:300] Negotiation successful
0430 20:01:27.886391 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":298,"threads_started":1}
W20260430 20:01:27.887923 10586 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49482
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 20:01:27 dist-test-slave-0bdw krb5kdc[10431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:27.923647 10951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:27.911334 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.10.46.1:41777)
0430 20:01:27.911702 (+ 368us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:27.911707 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:27.912122 (+ 415us) server_negotiation.cc:408] Connection header received
0430 20:01:27.912577 (+ 455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:27.912588 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:27.912661 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:27.912772 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:27.913811 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.914560 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:27.915761 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:27.916407 (+ 646us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:27.919261 (+ 2854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:27.919292 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:27.919294 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:27.919342 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:27.921011 (+ 1669us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.921636 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.921639 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.921641 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.921697 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:27.922132 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:27.922138 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:27.922139 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:27.922525 (+ 386us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:27.922749 (+ 224us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:27.923352 (+ 603us) server_negotiation.cc:300] Negotiation successful
0430 20:01:27.923496 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260430 20:01:27.925402 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 10543
I20260430 20:01:27.937202 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 10679
I20260430 20:01:27.946523 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 10815
I20260430 20:01:27.955986 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 10447
2026-04-30T20:01:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3627 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:28 dist-test-slave-0bdw krb5kdc[10958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:28 dist-test-slave-0bdw krb5kdc[10958](info): set up 2 sockets
Apr 30 20:01:28 dist-test-slave-0bdw krb5kdc[10958](info): commencing operation
krb5kdc: starting...
W20260430 20:01:30.153292 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.170s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:30 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:30.329298 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34311
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:30.459434 10974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:30.460243 10974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:30.460407 10974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:30.464680 10974 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:30.464787 10974 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:30.464812 10974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:30.464834 10974 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:30.464854 10974 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:30.473098 10974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:30.475433 10974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:30.476984 10974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:30.485917 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:30.485909 10980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:30.486387 10974 server_base.cc:1061] running on GCE node
W20260430 20:01:30.485909 10982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:30.487331 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:30.488610 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:30.489843 10974 hybrid_clock.cc:648] HybridClock initialized: now 1777579290489790 us; error 77 us; skew 500 ppm
Apr 30 20:01:30 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:30.493986 10974 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:30.496014 10974 webserver.cc:492] Webserver started at http://127.10.46.62:46729/ using document root <none> and password file <none>
I20260430 20:01:30.496922 10974 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:30.497022 10974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:30.497296 10974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:30.500629 10974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "9d30fd61ef91462a9cbad6407258b395"
format_stamp: "Formatted at 2026-04-30 20:01:30 on dist-test-slave-0bdw"
server_key: "15653a89aa3e2687212298eb60809e8f"
server_key_iv: "b0c2d49de9b1f48b7e9292cda5847ed2"
server_key_version: "encryptionkey@0"
I20260430 20:01:30.501375 10974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "9d30fd61ef91462a9cbad6407258b395"
format_stamp: "Formatted at 2026-04-30 20:01:30 on dist-test-slave-0bdw"
server_key: "15653a89aa3e2687212298eb60809e8f"
server_key_iv: "b0c2d49de9b1f48b7e9292cda5847ed2"
server_key_version: "encryptionkey@0"
I20260430 20:01:30.505556 10974 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260430 20:01:30.508761 10989 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:30.510478 10974 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260430 20:01:30.510757 10974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "9d30fd61ef91462a9cbad6407258b395"
format_stamp: "Formatted at 2026-04-30 20:01:30 on dist-test-slave-0bdw"
server_key: "15653a89aa3e2687212298eb60809e8f"
server_key_iv: "b0c2d49de9b1f48b7e9292cda5847ed2"
server_key_version: "encryptionkey@0"
I20260430 20:01:30.510933 10974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:30.546934 10974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:30.551651 10974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:30.552199 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:30.563898 10974 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:34311
I20260430 20:01:30.563913 11041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:34311 every 8 connection(s)
I20260430 20:01:30.566085 10974 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:30.567989 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 10974
I20260430 20:01:30.568122 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:30.568393 10424 external_mini_cluster.cc:1468] Setting key 3f4f10a380140cad0b08b2c14aaab4a5
I20260430 20:01:30.570323 11042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:30.577641 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395: Bootstrap starting.
Apr 30 20:01:30 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:30.580355 11042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:30.581291 11042 log.cc:826] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:30.584133 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395: No bootstrap required, opened a new log
I20260430 20:01:30.586992 11045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:30.570092 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.10.46.62:34311)
0430 20:01:30.570565 (+ 473us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:30.570578 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:30.570618 (+ 40us) server_negotiation.cc:408] Connection header received
0430 20:01:30.571495 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:30.571520 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:30.571946 (+ 426us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:30.572334 (+ 388us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:30.574223 (+ 1889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:30.575424 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:30.577463 (+ 2039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:30.577768 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:30.580876 (+ 3108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:30.580899 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:30.580916 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:30.580959 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:30.583703 (+ 2744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:30.584309 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:30.584315 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:30.584321 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:30.584411 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:30.584734 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:30.584738 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:30.584740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:30.585178 (+ 438us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:30.585374 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:30.585890 (+ 516us) server_negotiation.cc:300] Negotiation successful
0430 20:01:30.586177 (+ 287us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260430 20:01:30.587738 11042 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } }
I20260430 20:01:30.587996 11042 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:30.588037 11042 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d30fd61ef91462a9cbad6407258b395, State: Initialized, Role: FOLLOWER
I20260430 20:01:30.588553 11042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } }
I20260430 20:01:30.588709 11042 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:30.588759 11042 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:30.588838 11042 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:30.669963 11042 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } }
I20260430 20:01:30.670513 11042 leader_election.cc:304] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d30fd61ef91462a9cbad6407258b395; no voters:
I20260430 20:01:30.671452 11042 leader_election.cc:290] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:30.671857 11047 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:30.672123 11047 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [term 1 LEADER]: Becoming Leader. State: Replica: 9d30fd61ef91462a9cbad6407258b395, State: Running, Role: LEADER
I20260430 20:01:30.672536 11047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } }
I20260430 20:01:30.673538 11042 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:30.674574 11049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d30fd61ef91462a9cbad6407258b395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } } }
I20260430 20:01:30.674727 11049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:30.674813 11048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d30fd61ef91462a9cbad6407258b395. Latest consensus state: current_term: 1 leader_uuid: "9d30fd61ef91462a9cbad6407258b395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d30fd61ef91462a9cbad6407258b395" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34311 } } }
I20260430 20:01:30.674937 11048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:30.675174 11051 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:30.679476 11051 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:30.686050 11051 catalog_manager.cc:1357] Generated new cluster ID: b7fc92af368c46b0b7422a370dfe2cdd
I20260430 20:01:30.686159 11051 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:30.728557 11051 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:30.730062 11051 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:30.741128 11051 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9d30fd61ef91462a9cbad6407258b395: Generated new TSK 0
I20260430 20:01:30.741849 11051 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
W20260430 20:01:30.875943 10424 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.10.46.1: real 0.129s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:30.905339 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34311
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:31.031095 11070 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:31.031471 11070 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:31.031540 11070 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:31.035228 11070 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:31.035326 11070 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:31.035415 11070 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:31.041303 11070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:31.042703 11070 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:31.043646 11070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:31.052661 11078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:31.052690 11075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:31.053512 11070 server_base.cc:1061] running on GCE node
W20260430 20:01:31.053582 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:31.054219 11070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:31.055032 11070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:31.058416 11070 hybrid_clock.cc:648] HybridClock initialized: now 1777579291058289 us; error 154 us; skew 500 ppm
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:31.062096 11070 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:31.064335 11070 webserver.cc:492] Webserver started at http://127.10.46.1:45427/ using document root <none> and password file <none>
I20260430 20:01:31.065032 11070 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:31.065119 11070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:31.065351 11070 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:31.067422 11070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "f3ec73e754a7409da23fa3418c4ac43d"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "ce28764cc961e7d73daf254f1a7318dc"
server_key_iv: "3ad5c1090916d8b616fc121f2d57754c"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.068053 11070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "f3ec73e754a7409da23fa3418c4ac43d"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "ce28764cc961e7d73daf254f1a7318dc"
server_key_iv: "3ad5c1090916d8b616fc121f2d57754c"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.072557 11070 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:01:31.075941 11085 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.077651 11070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:01:31.077847 11070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "f3ec73e754a7409da23fa3418c4ac43d"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "ce28764cc961e7d73daf254f1a7318dc"
server_key_iv: "3ad5c1090916d8b616fc121f2d57754c"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.077992 11070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:31.096179 11070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:31.100337 11070 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:31.100574 11070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:31.101265 11070 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:31.102243 11070 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:31.102319 11070 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.102394 11070 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:31.102444 11070 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.116894 11070 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:38781
I20260430 20:01:31.117005 11198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:38781 every 8 connection(s)
I20260430 20:01:31.118077 11070 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:31.124362 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11070
I20260430 20:01:31.124509 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:31.124733 10424 external_mini_cluster.cc:1468] Setting key e4025c66e34bcdfd17850f65305932f6
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:01:31.138588 11201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.120496 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:36685 (local address 127.10.46.62:34311)
0430 20:01:31.120834 (+ 338us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.120838 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.121687 (+ 849us) server_negotiation.cc:408] Connection header received
0430 20:01:31.122629 (+ 942us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.122633 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.122696 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.122814 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.124654 (+ 1840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.125347 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.126036 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.126222 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.129718 (+ 3496us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.129769 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.129775 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.129818 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.134439 (+ 4621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.136152 (+ 1713us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.136157 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.136159 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.136382 (+ 223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.137253 (+ 871us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.137258 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.137260 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.137504 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.137641 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.138216 (+ 575us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.138402 (+ 186us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260430 20:01:31.139274 11202 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.120897 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34311 (local address 127.10.46.1:36685)
0430 20:01:31.121525 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.121562 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.122416 (+ 854us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.122982 (+ 566us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.122991 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.123409 (+ 418us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:31.124412 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.124428 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.125481 (+ 1053us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.125487 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.125895 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.125902 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.126162 (+ 260us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.126903 (+ 741us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:31.126925 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:31.129424 (+ 2499us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:31.134971 (+ 5547us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.134995 (+ 24us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.135013 (+ 18us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.135609 (+ 596us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.136802 (+ 1193us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.136818 (+ 16us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.136820 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.137004 (+ 184us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.137645 (+ 641us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:31.137655 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:31.137921 (+ 266us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.138158 (+ 237us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.138430 (+ 272us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":249,"threads_started":1}
I20260430 20:01:31.141194 11199 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34311
I20260430 20:01:31.141462 11199 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:31.142004 11199 heartbeater.cc:507] Master 127.10.46.62:34311 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:01:31.144142 11006 ts_manager.cc:194] Registered new tserver with Master: f3ec73e754a7409da23fa3418c4ac43d (127.10.46.1:38781)
I20260430 20:01:31.145534 11006 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:36685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:31.201138 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34311
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:31.324602 11207 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:31.324859 11207 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:31.324921 11207 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:31.328456 11207 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:31.328651 11207 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:31.328773 11207 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:31.334158 11207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:31.335345 11207 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:31.336287 11207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:31.343545 11213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:31.343545 11212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:31.343865 11207 server_base.cc:1061] running on GCE node
W20260430 20:01:31.343559 11215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:31.344486 11207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:31.345120 11207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:31.346365 11207 hybrid_clock.cc:648] HybridClock initialized: now 1777579291346337 us; error 44 us; skew 500 ppm
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:31.350960 11207 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:31.352715 11207 webserver.cc:492] Webserver started at http://127.10.46.2:33287/ using document root <none> and password file <none>
I20260430 20:01:31.353693 11207 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:31.353889 11207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:31.354198 11207 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:31.356866 11207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "91f6ca690056409bb63b4c3f320b9b64"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "238e76506f58d0214c9af4a87af8f5f9"
server_key_iv: "1020e65fc094fda348134df470f8fa6d"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.357419 11207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "91f6ca690056409bb63b4c3f320b9b64"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "238e76506f58d0214c9af4a87af8f5f9"
server_key_iv: "1020e65fc094fda348134df470f8fa6d"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.361826 11207 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260430 20:01:31.365700 11222 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.367044 11207 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 20:01:31.367205 11207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "91f6ca690056409bb63b4c3f320b9b64"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "238e76506f58d0214c9af4a87af8f5f9"
server_key_iv: "1020e65fc094fda348134df470f8fa6d"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.367374 11207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:31.391606 11207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:31.397003 11207 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:31.397323 11207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:31.398001 11207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:31.399169 11207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:31.399240 11207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.399286 11207 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:31.399362 11207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.414225 11207 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:45373
I20260430 20:01:31.414232 11335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:45373 every 8 connection(s)
I20260430 20:01:31.416146 11207 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:31.420585 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11207
I20260430 20:01:31.420727 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:31.420989 10424 external_mini_cluster.cc:1468] Setting key 09a45c7a4572fa0b66b0de8250d2dfd3
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:31.436442 11201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.418807 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:44371 (local address 127.10.46.62:34311)
0430 20:01:31.418982 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.418987 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.420460 (+ 1473us) server_negotiation.cc:408] Connection header received
0430 20:01:31.421634 (+ 1174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.421640 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.421715 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.421804 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.424052 (+ 2248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.424565 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.425790 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.426060 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.429712 (+ 3652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.429751 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.429757 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.429796 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.432522 (+ 2726us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.433648 (+ 1126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.433655 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.433660 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.433961 (+ 301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.434479 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.434488 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.434495 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.434971 (+ 476us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.435137 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.436134 (+ 997us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.436289 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 20:01:31.437233 11338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.419180 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34311 (local address 127.10.46.2:44371)
0430 20:01:31.420175 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.420216 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.421303 (+ 1087us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.422035 (+ 732us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.422068 (+ 33us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.422528 (+ 460us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:31.423822 (+ 1294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.423844 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.424765 (+ 921us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.424772 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.425559 (+ 787us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.425581 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.425920 (+ 339us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.426750 (+ 830us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:31.426781 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:31.429417 (+ 2636us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:31.432885 (+ 3468us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.432913 (+ 28us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.432927 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.433331 (+ 404us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.434177 (+ 846us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.434180 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.434182 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.434311 (+ 129us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.435194 (+ 883us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:31.435202 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:31.435564 (+ 362us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.436028 (+ 464us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.436305 (+ 277us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
I20260430 20:01:31.439059 11336 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34311
I20260430 20:01:31.439599 11336 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:31.440661 11336 heartbeater.cc:507] Master 127.10.46.62:34311 requested a full tablet report, sending...
I20260430 20:01:31.442306 11006 ts_manager.cc:194] Registered new tserver with Master: 91f6ca690056409bb63b4c3f320b9b64 (127.10.46.2:45373)
I20260430 20:01:31.443212 11006 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:44371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:31.489233 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34311
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:31.624348 11343 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:31.624612 11343 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:31.624677 11343 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:31.628571 11343 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:31.628665 11343 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:31.628753 11343 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:31.635396 11343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:31.637050 11343 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:31.638036 11343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:31.646212 11348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:31.646226 11351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:31.646214 11349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:31.646672 11343 server_base.cc:1061] running on GCE node
I20260430 20:01:31.647437 11343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:31.648572 11343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:31.649820 11343 hybrid_clock.cc:648] HybridClock initialized: now 1777579291649779 us; error 63 us; skew 500 ppm
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:31.653478 11343 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:31.654945 11343 webserver.cc:492] Webserver started at http://127.10.46.3:42683/ using document root <none> and password file <none>
I20260430 20:01:31.655685 11343 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:31.655752 11343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:31.656260 11343 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:31.659176 11343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "0b131f1cee654b0da755b4a6613c9cf1"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "831587a4fd908de462f426beba48bf0b"
server_key_iv: "68a04afa05b238eba030d56925af292b"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.660236 11343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "0b131f1cee654b0da755b4a6613c9cf1"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "831587a4fd908de462f426beba48bf0b"
server_key_iv: "68a04afa05b238eba030d56925af292b"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.664801 11343 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260430 20:01:31.668226 11358 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.669916 11343 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260430 20:01:31.670058 11343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "0b131f1cee654b0da755b4a6613c9cf1"
format_stamp: "Formatted at 2026-04-30 20:01:31 on dist-test-slave-0bdw"
server_key: "831587a4fd908de462f426beba48bf0b"
server_key_iv: "68a04afa05b238eba030d56925af292b"
server_key_version: "encryptionkey@0"
I20260430 20:01:31.670356 11343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:31.697387 11343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:31.702016 11343 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:31.702303 11343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:31.703001 11343 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:31.704137 11343 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:31.704190 11343 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.704234 11343 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:31.704267 11343 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:31.717968 11343 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:33351
I20260430 20:01:31.717973 11471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:33351 every 8 connection(s)
I20260430 20:01:31.719179 11343 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:31.728060 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11343
I20260430 20:01:31.728310 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:31.728652 10424 external_mini_cluster.cc:1468] Setting key a93fad8ed7baa7ce48de0c9490629521
I20260430 20:01:31.734931 11201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.721168 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:43079 (local address 127.10.46.62:34311)
0430 20:01:31.721338 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.721348 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.722086 (+ 738us) server_negotiation.cc:408] Connection header received
0430 20:01:31.722959 (+ 873us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.722963 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.723025 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.723137 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.724814 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.725335 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.725980 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.726177 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.729626 (+ 3449us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.729654 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.729657 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.729698 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.731972 (+ 2274us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.733294 (+ 1322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.733298 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.733300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.733374 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.733733 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.733737 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.733740 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.733917 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.734026 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.734592 (+ 566us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.734750 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 20:01:31.735723 11474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.721441 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34311 (local address 127.10.46.3:43079)
0430 20:01:31.721909 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.721944 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.722769 (+ 825us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.723294 (+ 525us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.723302 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.723700 (+ 398us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:31.724653 (+ 953us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.724667 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.725472 (+ 805us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.725476 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.725864 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.725870 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.726111 (+ 241us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.726904 (+ 793us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:31.726934 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:31.729389 (+ 2455us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:31.732151 (+ 2762us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.732160 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.732176 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.732805 (+ 629us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.733492 (+ 687us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:31.733496 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:31.733498 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:31.733630 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:31.734070 (+ 440us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:31.734077 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:31.734334 (+ 257us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.734556 (+ 222us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.734836 (+ 280us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260430 20:01:31.737455 11472 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34311
I20260430 20:01:31.737728 11472 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:31.738312 11472 heartbeater.cc:507] Master 127.10.46.62:34311 requested a full tablet report, sending...
I20260430 20:01:31.739457 11006 ts_manager.cc:194] Registered new tserver with Master: 0b131f1cee654b0da755b4a6613c9cf1 (127.10.46.3:33351)
I20260430 20:01:31.740103 11006 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:43079
I20260430 20:01:31.744489 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:01:31.756055 11201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.747932 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.10.46.62:34311)
0430 20:01:31.748154 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.748162 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.748421 (+ 259us) server_negotiation.cc:408] Connection header received
0430 20:01:31.748604 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.748608 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.748676 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.748871 (+ 195us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.749779 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.750287 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.751020 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.751222 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.752463 (+ 1241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.752494 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.752497 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.752522 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.754398 (+ 1876us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.754854 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.754857 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.754858 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.754903 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.755150 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.755154 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.755156 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.755341 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.755495 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.755721 (+ 226us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.755885 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260430 20:01:31.763550 11006 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:01:31.766388 11006 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:31.786770 11486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.781256 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:45373 (local address 127.0.0.1:40260)
0430 20:01:31.782316 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.782335 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.782446 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.782941 (+ 495us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.782947 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.782969 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:31.783227 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.783234 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.785082 (+ 1848us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.785089 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.786202 (+ 1113us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.786212 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.786341 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.786409 (+ 68us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.786503 (+ 94us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.786583 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":127,"threads_started":1}
I20260430 20:01:31.787006 11488 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.782559 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38781 (local address 127.0.0.1:53844)
0430 20:01:31.782835 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.782853 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.782942 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.783708 (+ 766us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.783714 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.783759 (+ 45us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:31.784109 (+ 350us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.784118 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.785637 (+ 1519us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.785641 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.786595 (+ 954us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.786603 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.786715 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.786729 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.786781 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.786845 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260430 20:01:31.787388 11487 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.781983 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.10.46.3:33351)
0430 20:01:31.782597 (+ 614us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.782605 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.782627 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:01:31.783026 (+ 399us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.783035 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.783198 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.783358 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:31.784031 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.785382 (+ 1351us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.786357 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.786894 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.787023 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.787091 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.787188 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":158,"threads_started":1}
I20260430 20:01:31.787671 11484 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.781501 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:33351 (local address 127.0.0.1:52172)
0430 20:01:31.782121 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.782162 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.782309 (+ 147us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.783548 (+ 1239us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.783554 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.783578 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:31.783875 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.783886 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.785520 (+ 1634us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.785524 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.786182 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.786190 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.786338 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.786409 (+ 71us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.786507 (+ 98us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.786586 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":105,"threads_started":1}
I20260430 20:01:31.788555 11489 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.782753 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.10.46.1:38781)
0430 20:01:31.783190 (+ 437us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.783197 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.783218 (+ 21us) server_negotiation.cc:408] Connection header received
0430 20:01:31.783305 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.783310 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.783464 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.783606 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:31.784378 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.785420 (+ 1042us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.787355 (+ 1935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.788090 (+ 735us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.788216 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.788300 (+ 84us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.788407 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":93,"threads_started":1}
I20260430 20:01:31.788906 11485 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.781389 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.10.46.2:45373)
0430 20:01:31.781834 (+ 445us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.781841 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.782474 (+ 633us) server_negotiation.cc:408] Connection header received
0430 20:01:31.782573 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.782578 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.782773 (+ 195us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.782927 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:31.783373 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.784901 (+ 1528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.786691 (+ 1790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.788504 (+ 1813us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.788599 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.788673 (+ 74us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.788763 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":147,"threads_started":1}
I20260430 20:01:31.791566 11133 tablet_service.cc:1511] Processing CreateTablet for tablet c31a8b4d3fee45adad808ba2790996e4 (DEFAULT_TABLE table=test-table [id=dc931e86ec0a41ba9217d12fb945f0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:31.791566 11270 tablet_service.cc:1511] Processing CreateTablet for tablet c31a8b4d3fee45adad808ba2790996e4 (DEFAULT_TABLE table=test-table [id=dc931e86ec0a41ba9217d12fb945f0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:31.792368 11406 tablet_service.cc:1511] Processing CreateTablet for tablet c31a8b4d3fee45adad808ba2790996e4 (DEFAULT_TABLE table=test-table [id=dc931e86ec0a41ba9217d12fb945f0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:31.792742 11270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31a8b4d3fee45adad808ba2790996e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:31.792923 11133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31a8b4d3fee45adad808ba2790996e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:31.793905 11406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31a8b4d3fee45adad808ba2790996e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:31.799613 11490 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Bootstrap starting.
I20260430 20:01:31.800339 11491 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Bootstrap starting.
I20260430 20:01:31.800961 11492 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Bootstrap starting.
I20260430 20:01:31.802098 11490 tablet_bootstrap.cc:654] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:31.802426 11491 tablet_bootstrap.cc:654] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:31.802860 11490 log.cc:826] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:31.802930 11492 tablet_bootstrap.cc:654] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:31.803208 11491 log.cc:826] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:31.803694 11492 log.cc:826] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:31.805133 11490 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: No bootstrap required, opened a new log
I20260430 20:01:31.805377 11490 ts_tablet_manager.cc:1403] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 20:01:31.805392 11491 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: No bootstrap required, opened a new log
I20260430 20:01:31.805384 11492 tablet_bootstrap.cc:492] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: No bootstrap required, opened a new log
I20260430 20:01:31.805560 11491 ts_tablet_manager.cc:1403] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:01:31.805560 11492 ts_tablet_manager.cc:1403] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:01:31.808084 11490 raft_consensus.cc:359] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.808310 11490 raft_consensus.cc:385] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:31.808373 11490 raft_consensus.cc:740] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ec73e754a7409da23fa3418c4ac43d, State: Initialized, Role: FOLLOWER
I20260430 20:01:31.808800 11490 consensus_queue.cc:260] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.809487 11199 heartbeater.cc:499] Master 127.10.46.62:34311 was elected leader, sending a full tablet report...
I20260430 20:01:31.809430 11492 raft_consensus.cc:359] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.809430 11491 raft_consensus.cc:359] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.809629 11491 raft_consensus.cc:385] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:31.809629 11492 raft_consensus.cc:385] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:31.809679 11491 raft_consensus.cc:740] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f6ca690056409bb63b4c3f320b9b64, State: Initialized, Role: FOLLOWER
I20260430 20:01:31.809679 11492 raft_consensus.cc:740] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b131f1cee654b0da755b4a6613c9cf1, State: Initialized, Role: FOLLOWER
I20260430 20:01:31.809755 11490 ts_tablet_manager.cc:1434] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260430 20:01:31.810127 11491 consensus_queue.cc:260] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.810127 11492 consensus_queue.cc:260] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.810958 11492 ts_tablet_manager.cc:1434] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:01:31.810964 11336 heartbeater.cc:499] Master 127.10.46.62:34311 was elected leader, sending a full tablet report...
I20260430 20:01:31.811002 11472 heartbeater.cc:499] Master 127.10.46.62:34311 was elected leader, sending a full tablet report...
I20260430 20:01:31.814369 11491 ts_tablet_manager.cc:1434] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20260430 20:01:31.856683 11497 raft_consensus.cc:493] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:31.857117 11497 raft_consensus.cc:515] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.858484 11497 leader_election.cc:290] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b131f1cee654b0da755b4a6613c9cf1 (127.10.46.3:33351), f3ec73e754a7409da23fa3418c4ac43d (127.10.46.1:38781)
I20260430 20:01:31.862521 11499 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.858886 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:33351 (local address 127.10.46.2:43987)
0430 20:01:31.859273 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.859285 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.859372 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.859812 (+ 440us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.859818 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.859867 (+ 49us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:31.860264 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.860270 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.861283 (+ 1013us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.861287 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.862030 (+ 743us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.862041 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.862177 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.862196 (+ 19us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.862243 (+ 47us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.862298 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"spinlock_wait_cycles":384,"thread_start_us":136,"threads_started":1}
I20260430 20:01:31.862882 11338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.858845 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38781 (local address 127.10.46.2:59577)
0430 20:01:31.859126 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:31.859142 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:31.859261 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:31.859812 (+ 551us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:31.859818 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:31.859871 (+ 53us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:31.860260 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.860270 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.861508 (+ 1238us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.861511 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:31.862258 (+ 747us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:31.862269 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.862434 (+ 165us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.862456 (+ 22us) client_negotiation.cc:770] Sending connection context
0430 20:01:31.862514 (+ 58us) client_negotiation.cc:241] Negotiation successful
0430 20:01:31.862594 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"spinlock_wait_cycles":16128}
I20260430 20:01:31.863343 11489 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.858967 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:59577 (local address 127.10.46.1:38781)
0430 20:01:31.859168 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.859174 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.859199 (+ 25us) server_negotiation.cc:408] Connection header received
0430 20:01:31.859437 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.859441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.859499 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.859596 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:31.860545 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.861322 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.862459 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.863051 (+ 592us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.863097 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.863170 (+ 73us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.863235 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260430 20:01:31.863482 11487 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.858950 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:43987 (local address 127.10.46.3:33351)
0430 20:01:31.859092 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.859097 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.859388 (+ 291us) server_negotiation.cc:408] Connection header received
0430 20:01:31.859443 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.859446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.859499 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.859595 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:31.860469 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.861108 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.862288 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.863135 (+ 847us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.863188 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.863284 (+ 96us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.863349 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260430 20:01:31.864176 11426 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31a8b4d3fee45adad808ba2790996e4" candidate_uuid: "91f6ca690056409bb63b4c3f320b9b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b131f1cee654b0da755b4a6613c9cf1" is_pre_election: true
I20260430 20:01:31.864398 11153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31a8b4d3fee45adad808ba2790996e4" candidate_uuid: "91f6ca690056409bb63b4c3f320b9b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ec73e754a7409da23fa3418c4ac43d" is_pre_election: true
I20260430 20:01:31.864514 11426 raft_consensus.cc:2468] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f6ca690056409bb63b4c3f320b9b64 in term 0.
I20260430 20:01:31.864675 11153 raft_consensus.cc:2468] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f6ca690056409bb63b4c3f320b9b64 in term 0.
I20260430 20:01:31.865217 11224 leader_election.cc:304] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b131f1cee654b0da755b4a6613c9cf1, 91f6ca690056409bb63b4c3f320b9b64; no voters:
I20260430 20:01:31.865511 11497 raft_consensus.cc:2804] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:31.865651 11497 raft_consensus.cc:493] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:31.865748 11497 raft_consensus.cc:3060] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:31.866798 11497 raft_consensus.cc:515] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.867242 11497 leader_election.cc:290] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [CANDIDATE]: Term 1 election: Requested vote from peers 0b131f1cee654b0da755b4a6613c9cf1 (127.10.46.3:33351), f3ec73e754a7409da23fa3418c4ac43d (127.10.46.1:38781)
I20260430 20:01:31.867659 11426 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31a8b4d3fee45adad808ba2790996e4" candidate_uuid: "91f6ca690056409bb63b4c3f320b9b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b131f1cee654b0da755b4a6613c9cf1"
I20260430 20:01:31.867930 11426 raft_consensus.cc:3060] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:31.868144 11153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31a8b4d3fee45adad808ba2790996e4" candidate_uuid: "91f6ca690056409bb63b4c3f320b9b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ec73e754a7409da23fa3418c4ac43d"
I20260430 20:01:31.868294 11153 raft_consensus.cc:3060] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:31.869129 11426 raft_consensus.cc:2468] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f6ca690056409bb63b4c3f320b9b64 in term 1.
I20260430 20:01:31.869679 11224 leader_election.cc:304] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b131f1cee654b0da755b4a6613c9cf1, 91f6ca690056409bb63b4c3f320b9b64; no voters:
W20260430 20:01:31.869910 11200 tablet.cc:2404] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:01:31.870301 11153 raft_consensus.cc:2468] T c31a8b4d3fee45adad808ba2790996e4 P f3ec73e754a7409da23fa3418c4ac43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f6ca690056409bb63b4c3f320b9b64 in term 1.
I20260430 20:01:31.870685 11497 raft_consensus.cc:2804] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:31.871171 11497 raft_consensus.cc:697] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [term 1 LEADER]: Becoming Leader. State: Replica: 91f6ca690056409bb63b4c3f320b9b64, State: Running, Role: LEADER
I20260430 20:01:31.871742 11497 consensus_queue.cc:237] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } }
I20260430 20:01:31.876942 11006 catalog_manager.cc:5671] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91f6ca690056409bb63b4c3f320b9b64 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "91f6ca690056409bb63b4c3f320b9b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f6ca690056409bb63b4c3f320b9b64" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 45373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b131f1cee654b0da755b4a6613c9cf1" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ec73e754a7409da23fa3418c4ac43d" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38781 } health_report { overall_health: UNKNOWN } } }
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:31.891948 11489 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.882227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.10.46.1:38781)
0430 20:01:31.882396 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.882400 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.882512 (+ 112us) server_negotiation.cc:408] Connection header received
0430 20:01:31.882674 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.882677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.882733 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.882829 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.883660 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.884236 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.884938 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.885158 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.888189 (+ 3031us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.888218 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.888230 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.888261 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.890194 (+ 1933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.890691 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.890696 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.890698 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.890749 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.890991 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.890994 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.890996 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.891190 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.891394 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.891604 (+ 210us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.891743 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260430 20:01:31.918385 11337 tablet.cc:2404] T c31a8b4d3fee45adad808ba2790996e4 P 91f6ca690056409bb63b4c3f320b9b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:31.924896 11489 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.913270 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.10.46.1:38781)
0430 20:01:31.913555 (+ 285us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.913561 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.913805 (+ 244us) server_negotiation.cc:408] Connection header received
0430 20:01:31.914008 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.914011 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.914061 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.914153 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.915046 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.915545 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.916293 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.916518 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.920799 (+ 4281us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.920829 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.920831 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.920868 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.922579 (+ 1711us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.923059 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.923062 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.923064 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.923139 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.923613 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.923616 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.923618 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.923839 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.924254 (+ 415us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.924548 (+ 294us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.924667 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 20:01:31 dist-test-slave-0bdw krb5kdc[10958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:31.957870 11489 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:31.947683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.10.46.1:38781)
0430 20:01:31.947938 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:31.947942 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:31.947962 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:01:31.948027 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:31.948030 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:31.948090 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:31.948230 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:31.949383 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.949869 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:31.950587 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:31.950728 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:31.953868 (+ 3140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:31.953900 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:31.953903 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:31.953945 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:31.955537 (+ 1592us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.956360 (+ 823us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.956365 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.956367 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.956536 (+ 169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:31.956825 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:31.956827 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:31.956829 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:31.957051 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:31.957200 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:31.957502 (+ 302us) server_negotiation.cc:300] Negotiation successful
0430 20:01:31.957660 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260430 20:01:31.959673 11111 tablet_service.cc:3018] Rejecting scan request for tablet c31a8b4d3fee45adad808ba2790996e4: Uninitialized: clean time has not yet been initialized
I20260430 20:01:31.962208 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11070
I20260430 20:01:31.968582 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11207
W20260430 20:01:31.970237 11473 tablet.cc:2404] T c31a8b4d3fee45adad808ba2790996e4 P 0b131f1cee654b0da755b4a6613c9cf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:01:31.975436 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11343
I20260430 20:01:31.983358 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 10974
2026-04-30T20:01:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4026 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:32 dist-test-slave-0bdw krb5kdc[11514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:32 dist-test-slave-0bdw krb5kdc[11514](info): set up 2 sockets
Apr 30 20:01:32 dist-test-slave-0bdw krb5kdc[11514](info): commencing operation
krb5kdc: starting...
W20260430 20:01:34.080647 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.071s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:34.245251 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41819
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:34.356618 11530 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:34.357105 11530 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:34.357322 11530 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:34.361299 11530 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:34.361379 11530 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 20:01:34.361399 11530 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:34.361423 11530 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:34.361471 11530 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:34.361507 11530 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:34.366514 11530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:41819
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:34.367918 11530 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:34.368853 11530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:34.375727 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:34.375727 11536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:34.375728 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:34.376070 11530 server_base.cc:1061] running on GCE node
I20260430 20:01:34.376861 11530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:34.377904 11530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:34.379106 11530 hybrid_clock.cc:648] HybridClock initialized: now 1777579294379078 us; error 44 us; skew 500 ppm
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:34.382454 11530 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:34.383687 11530 webserver.cc:492] Webserver started at http://127.10.46.62:45049/ using document root <none> and password file <none>
I20260430 20:01:34.384337 11530 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:34.384393 11530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:34.384560 11530 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:34.386374 11530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "29e06c6e82374a46a3f524d95865a257"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "f3c6d156c931e560bc25369a10a3f301"
server_key_iv: "a992ba08f80031aef201f9da537ef02b"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.386848 11530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "29e06c6e82374a46a3f524d95865a257"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "f3c6d156c931e560bc25369a10a3f301"
server_key_iv: "a992ba08f80031aef201f9da537ef02b"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.390606 11530 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:01:34.393854 11545 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:34.394968 11530 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:01:34.395146 11530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "29e06c6e82374a46a3f524d95865a257"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "f3c6d156c931e560bc25369a10a3f301"
server_key_iv: "a992ba08f80031aef201f9da537ef02b"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.395272 11530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:34.420266 11530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:34.423702 11530 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:34.424154 11530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:34.435338 11530 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:41819
I20260430 20:01:34.435644 11607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:41819 every 8 connection(s)
I20260430 20:01:34.436594 11530 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:34.439481 11608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:34.442116 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11530
I20260430 20:01:34.442239 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:34.442485 10424 external_mini_cluster.cc:1468] Setting key d9ecfb7ce31bcf4a960f1cb03a89d92b
I20260430 20:01:34.445158 11608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Bootstrap starting.
I20260430 20:01:34.447932 11608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:34.448763 11608 log.cc:826] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:34.451016 11608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: No bootstrap required, opened a new log
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:34.453924 11608 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } }
I20260430 20:01:34.454128 11608 raft_consensus.cc:385] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:34.454164 11608 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e06c6e82374a46a3f524d95865a257, State: Initialized, Role: FOLLOWER
I20260430 20:01:34.454671 11608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } }
I20260430 20:01:34.454842 11608 raft_consensus.cc:399] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:34.454921 11608 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:34.455044 11608 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:34.456348 11608 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } }
I20260430 20:01:34.456772 11608 leader_election.cc:304] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e06c6e82374a46a3f524d95865a257; no voters:
I20260430 20:01:34.457134 11608 leader_election.cc:290] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:34.457341 11613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:34.457648 11613 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [term 1 LEADER]: Becoming Leader. State: Replica: 29e06c6e82374a46a3f524d95865a257, State: Running, Role: LEADER
I20260430 20:01:34.458104 11613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } }
I20260430 20:01:34.458454 11608 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:34.459746 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e06c6e82374a46a3f524d95865a257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } } }
I20260430 20:01:34.459983 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:34.461150 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:34.444161 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.10.46.62:41819)
0430 20:01:34.444622 (+ 461us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:34.444631 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:34.444746 (+ 115us) server_negotiation.cc:408] Connection header received
0430 20:01:34.445470 (+ 724us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:34.445487 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:34.445765 (+ 278us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:34.446042 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:34.447159 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.448615 (+ 1456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:34.449581 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.449970 (+ 389us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:34.454478 (+ 4508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:34.454506 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:34.454521 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:34.454561 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:34.457493 (+ 2932us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.458133 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.458139 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.458145 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.458250 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.458536 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.458540 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.458542 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.458962 (+ 420us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:34.459162 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:34.460052 (+ 890us) server_negotiation.cc:300] Negotiation successful
0430 20:01:34.460300 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":167,"threads_started":1}
I20260430 20:01:34.461436 11615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e06c6e82374a46a3f524d95865a257. Latest consensus state: current_term: 1 leader_uuid: "29e06c6e82374a46a3f524d95865a257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e06c6e82374a46a3f524d95865a257" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41819 } } }
I20260430 20:01:34.462308 11615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257 [sys.catalog]: This master's current role is: LEADER
W20260430 20:01:34.464365 11628 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:01:34.464452 11628 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:01:34.464565 11622 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:34.465355 11622 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:34.471271 11622 catalog_manager.cc:1357] Generated new cluster ID: 49ba7c202cfd42cfaa7e78947acc08c1
I20260430 20:01:34.471364 11622 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:34.479590 11622 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:34.480798 11622 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:34.500196 11622 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Generated new TSK 0
I20260430 20:01:34.500938 11622 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:34.576509 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41819
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:34.687364 11636 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:34.687667 11636 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:34.687762 11636 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:34.691282 11636 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:34.691367 11636 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:34.691423 11636 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:34.691475 11636 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:34.696179 11636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:34.697392 11636 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:34.698256 11636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:34.705312 11641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:34.705317 11642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:34.705313 11644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:34.705860 11636 server_base.cc:1061] running on GCE node
I20260430 20:01:34.706588 11636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:34.707947 11636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:34.709201 11636 hybrid_clock.cc:648] HybridClock initialized: now 1777579294709198 us; error 68 us; skew 500 ppm
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:34.712486 11636 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:34.713987 11636 webserver.cc:492] Webserver started at http://127.10.46.1:36321/ using document root <none> and password file <none>
I20260430 20:01:34.714906 11636 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:34.714998 11636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:34.715245 11636 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:34.718022 11636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "3d730c5d154b4d1e82b2e59cc181c640"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "51894259fa305e2101ae413fa5025e77"
server_key_iv: "dad842936cd341740f36dd2fe0e3a61e"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.719014 11636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "3d730c5d154b4d1e82b2e59cc181c640"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "51894259fa305e2101ae413fa5025e77"
server_key_iv: "dad842936cd341740f36dd2fe0e3a61e"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.723639 11636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:01:34.726481 11651 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:34.727658 11636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:34.727835 11636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "3d730c5d154b4d1e82b2e59cc181c640"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "51894259fa305e2101ae413fa5025e77"
server_key_iv: "dad842936cd341740f36dd2fe0e3a61e"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.727960 11636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:34.750742 11636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:34.759464 11636 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:34.759713 11636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:34.761510 11636 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:34.761574 11636 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:34.761615 11636 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:34.761653 11636 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:34.773887 11636 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:46593
I20260430 20:01:34.774268 11766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:46593 every 8 connection(s)
I20260430 20:01:34.775113 11636 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:34.776592 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:34.762745 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:56177 (local address 127.10.46.62:41819)
0430 20:01:34.762885 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:34.762896 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:34.764020 (+ 1124us) server_negotiation.cc:408] Connection header received
0430 20:01:34.765073 (+ 1053us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:34.765078 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:34.765151 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:34.765246 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:34.766912 (+ 1666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.767830 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:34.768667 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.768852 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:34.771610 (+ 2758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:34.771641 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:34.771646 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:34.771682 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:34.774131 (+ 2449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.774725 (+ 594us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.774730 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.774731 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.774926 (+ 195us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.775313 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.775317 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.775319 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.775566 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:34.775676 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:34.776338 (+ 662us) server_negotiation.cc:300] Negotiation successful
0430 20:01:34.776460 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 20:01:34.777712 11660 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:34.763112 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.1:56177)
0430 20:01:34.763865 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:34.763912 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:34.764803 (+ 891us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:34.765476 (+ 673us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:34.765489 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:34.766011 (+ 522us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:34.766706 (+ 695us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:34.766737 (+ 31us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.767963 (+ 1226us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:34.767967 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:34.768547 (+ 580us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:34.768555 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.768774 (+ 219us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:34.769423 (+ 649us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:34.769446 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:34.771326 (+ 1880us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:34.774313 (+ 2987us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:34.774320 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:34.774332 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:34.774606 (+ 274us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:34.775063 (+ 457us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:34.775070 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:34.775076 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:34.775206 (+ 130us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:34.775732 (+ 526us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:34.775738 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:34.776044 (+ 306us) client_negotiation.cc:770] Sending connection context
0430 20:01:34.776290 (+ 246us) client_negotiation.cc:241] Negotiation successful
0430 20:01:34.776830 (+ 540us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":134,"threads_started":1}
I20260430 20:01:34.779558 11767 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41819
I20260430 20:01:34.779742 11767 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:34.780385 11767 heartbeater.cc:507] Master 127.10.46.62:41819 requested a full tablet report, sending...
I20260430 20:01:34.782610 11562 ts_manager.cc:194] Registered new tserver with Master: 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593)
I20260430 20:01:34.784121 11562 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:56177
I20260430 20:01:34.784353 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11636
I20260430 20:01:34.784474 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:34.784749 10424 external_mini_cluster.cc:1468] Setting key 7ba36873d01a740b2b846b158f28745d
I20260430 20:01:34.789620 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:34.782149 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:54343 (local address 127.10.46.62:41819)
0430 20:01:34.782281 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:34.782284 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:34.782532 (+ 248us) server_negotiation.cc:408] Connection header received
0430 20:01:34.782681 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:34.782685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:34.782753 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:34.782834 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:34.783581 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.784190 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:34.785056 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.785260 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:34.786334 (+ 1074us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:34.786352 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:34.786355 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:34.786381 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:34.788159 (+ 1778us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.788563 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.788568 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.788570 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.788622 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:34.788860 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:34.788864 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:34.788866 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:34.789010 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:34.789115 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:34.789363 (+ 248us) server_negotiation.cc:300] Negotiation successful
0430 20:01:34.789471 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260430 20:01:34.789635 11773 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:34.782053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.1:54343)
0430 20:01:34.782440 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:34.782454 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:34.782580 (+ 126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:34.782882 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:34.782885 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:34.783139 (+ 254us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:34.783433 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:34.783442 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.784358 (+ 916us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:34.784362 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:34.784944 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:34.784952 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:34.785057 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:34.785693 (+ 636us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:34.785709 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:34.786166 (+ 457us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:34.788280 (+ 2114us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:34.788284 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:34.788287 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:34.788467 (+ 180us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:34.788721 (+ 254us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:34.788723 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:34.788725 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:34.788770 (+ 45us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:34.789104 (+ 334us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:34.789107 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:34.789192 (+ 85us) client_negotiation.cc:770] Sending connection context
0430 20:01:34.789292 (+ 100us) client_negotiation.cc:241] Negotiation successful
0430 20:01:34.789447 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:34.844198 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41819
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:34.961140 11778 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:34.961499 11778 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:34.961570 11778 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:34.965165 11778 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:34.965255 11778 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:34.965314 11778 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:34.965394 11778 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:34.970507 11778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:34.972065 11778 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:34.973012 11778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:34.980705 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:34.980705 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:34.981163 11778 server_base.cc:1061] running on GCE node
W20260430 20:01:34.981326 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:34.981911 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:34.982605 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:34.983875 11778 hybrid_clock.cc:648] HybridClock initialized: now 1777579294983824 us; error 55 us; skew 500 ppm
Apr 30 20:01:34 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:34.986650 11778 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:34.987757 11778 webserver.cc:492] Webserver started at http://127.10.46.2:45423/ using document root <none> and password file <none>
I20260430 20:01:34.988456 11778 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:34.988530 11778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:34.988767 11778 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:34.990669 11778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "5182df9b9c1d4bc291ec250345706a6a"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "6ee0f691f3df4cf1fa491b96f0c23f5b"
server_key_iv: "2a8947ab93bb1425d4976fb35c0838b7"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.991186 11778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "5182df9b9c1d4bc291ec250345706a6a"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "6ee0f691f3df4cf1fa491b96f0c23f5b"
server_key_iv: "2a8947ab93bb1425d4976fb35c0838b7"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.994812 11778 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260430 20:01:34.997412 11793 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:34.998689 11778 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 20:01:34.998845 11778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "5182df9b9c1d4bc291ec250345706a6a"
format_stamp: "Formatted at 2026-04-30 20:01:34 on dist-test-slave-0bdw"
server_key: "6ee0f691f3df4cf1fa491b96f0c23f5b"
server_key_iv: "2a8947ab93bb1425d4976fb35c0838b7"
server_key_version: "encryptionkey@0"
I20260430 20:01:34.998970 11778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:35.018486 11778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:35.021788 11778 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:35.022011 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:35.023626 11778 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:35.023695 11778 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:35.023738 11778 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:35.023753 11778 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:35.037009 11778 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:40447
I20260430 20:01:35.037081 11908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:40447 every 8 connection(s)
I20260430 20:01:35.038534 11778 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:35.039391 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.025015 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:43109 (local address 127.10.46.62:41819)
0430 20:01:35.025179 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.025182 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.026095 (+ 913us) server_negotiation.cc:408] Connection header received
0430 20:01:35.027258 (+ 1163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.027264 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.027342 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.027453 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.029920 (+ 2467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.030552 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.031391 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.031568 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.034792 (+ 3224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.034817 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.034820 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.034851 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.036897 (+ 2046us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.037559 (+ 662us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.037563 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.037564 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.037625 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.038111 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.038115 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.038116 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.038293 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.038385 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.039046 (+ 661us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.039198 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260430 20:01:35.040519 11802 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.025436 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.2:43109)
0430 20:01:35.025918 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.025966 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.026944 (+ 978us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.027738 (+ 794us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.027753 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.028641 (+ 888us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:35.029574 (+ 933us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.029598 (+ 24us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.030719 (+ 1121us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.030740 (+ 21us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.031256 (+ 516us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.031265 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.031521 (+ 256us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.032265 (+ 744us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:35.032290 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:35.034601 (+ 2311us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:35.037066 (+ 2465us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.037074 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.037094 (+ 20us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.037416 (+ 322us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.037828 (+ 412us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.037831 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.037833 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.037962 (+ 129us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.038409 (+ 447us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:35.038415 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:35.038708 (+ 293us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.038946 (+ 238us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.039267 (+ 321us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260430 20:01:35.041155 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11778
I20260430 20:01:35.041306 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:35.041584 10424 external_mini_cluster.cc:1468] Setting key 44cadcbbd9f566dbd06331bcdae81571
I20260430 20:01:35.043853 11909 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41819
I20260430 20:01:35.044046 11909 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:35.044634 11909 heartbeater.cc:507] Master 127.10.46.62:41819 requested a full tablet report, sending...
I20260430 20:01:35.045692 11561 ts_manager.cc:194] Registered new tserver with Master: 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2:40447)
I20260430 20:01:35.046504 11561 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:43109
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:35.053165 11915 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.044489 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.2:37215)
0430 20:01:35.044953 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.044974 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.045155 (+ 181us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.045561 (+ 406us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.045567 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.045812 (+ 245us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:35.046058 (+ 246us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.046065 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.047114 (+ 1049us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.047121 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.047827 (+ 706us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.047837 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.047963 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.048540 (+ 577us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:35.048561 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:35.049026 (+ 465us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:35.051214 (+ 2188us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.051218 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.051221 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.051475 (+ 254us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.051801 (+ 326us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.051805 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.051808 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.051879 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.052356 (+ 477us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:35.052362 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:35.052472 (+ 110us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.052597 (+ 125us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.052711 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
I20260430 20:01:35.053545 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.044578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:37215 (local address 127.10.46.62:41819)
0430 20:01:35.044722 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.044727 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.045050 (+ 323us) server_negotiation.cc:408] Connection header received
0430 20:01:35.045270 (+ 220us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.045275 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.045336 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.045431 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.046177 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.046747 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.048043 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.048351 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.049320 (+ 969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.049336 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.049339 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.049363 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.051089 (+ 1726us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.051605 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.051608 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.051610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.051665 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.052004 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.052007 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.052008 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.052194 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.052332 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.052992 (+ 660us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.053264 (+ 272us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:35.100795 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41819
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:35.214305 11920 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:35.214568 11920 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:35.214629 11920 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:35.218338 11920 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:35.218425 11920 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:35.218482 11920 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:35.218525 11920 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:35.223129 11920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:35.224280 11920 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:35.225121 11920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:35.231990 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:35.231967 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:35.232157 11920 server_base.cc:1061] running on GCE node
W20260430 20:01:35.231967 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:35.232648 11920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:35.233265 11920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:35.234465 11920 hybrid_clock.cc:648] HybridClock initialized: now 1777579295234457 us; error 49 us; skew 500 ppm
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:35.237699 11920 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:35.238907 11920 webserver.cc:492] Webserver started at http://127.10.46.3:34737/ using document root <none> and password file <none>
I20260430 20:01:35.239507 11920 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:35.239583 11920 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:35.239970 11920 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:35.241848 11920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "1447a1e8db614370b857a74613dd2340"
format_stamp: "Formatted at 2026-04-30 20:01:35 on dist-test-slave-0bdw"
server_key: "0dd6383139152a5277172579d28a8d2f"
server_key_iv: "17582bd81fefed630028f412e3bb4ea5"
server_key_version: "encryptionkey@0"
I20260430 20:01:35.242385 11920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "1447a1e8db614370b857a74613dd2340"
format_stamp: "Formatted at 2026-04-30 20:01:35 on dist-test-slave-0bdw"
server_key: "0dd6383139152a5277172579d28a8d2f"
server_key_iv: "17582bd81fefed630028f412e3bb4ea5"
server_key_version: "encryptionkey@0"
I20260430 20:01:35.246500 11920 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260430 20:01:35.249722 11935 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:35.251020 11920 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:35.251186 11920 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "1447a1e8db614370b857a74613dd2340"
format_stamp: "Formatted at 2026-04-30 20:01:35 on dist-test-slave-0bdw"
server_key: "0dd6383139152a5277172579d28a8d2f"
server_key_iv: "17582bd81fefed630028f412e3bb4ea5"
server_key_version: "encryptionkey@0"
I20260430 20:01:35.251312 11920 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:35.267067 11920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:35.271337 11920 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:35.271525 11920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:35.273394 11920 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:35.273487 11920 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:35.273556 11920 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:35.273595 11920 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:35.285919 11920 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:32839
I20260430 20:01:35.285997 12050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:32839 every 8 connection(s)
I20260430 20:01:35.287185 11920 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:35.288031 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 11920
I20260430 20:01:35.288295 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:35.288612 10424 external_mini_cluster.cc:1468] Setting key 27fc121b133f00785d3d0f53f8a0a705
I20260430 20:01:35.290939 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.274545 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:49955 (local address 127.10.46.62:41819)
0430 20:01:35.274692 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.274696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.275658 (+ 962us) server_negotiation.cc:408] Connection header received
0430 20:01:35.276830 (+ 1172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.276835 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.276908 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.276997 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.278930 (+ 1933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.279763 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.280894 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.281110 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.285612 (+ 4502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.285643 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.285649 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.285698 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.287329 (+ 1631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.288578 (+ 1249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.288585 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.288587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.288676 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.289420 (+ 744us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.289424 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.289427 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.289677 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.289788 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.290626 (+ 838us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.290763 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 20:01:35.291580 11944 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.274942 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.3:49955)
0430 20:01:35.275503 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.275541 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.276567 (+ 1026us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.277462 (+ 895us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.277475 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.278012 (+ 537us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:35.278761 (+ 749us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.278776 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.279915 (+ 1139us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.279919 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.280552 (+ 633us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.280573 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.281022 (+ 449us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.282120 (+ 1098us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:35.282173 (+ 53us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:35.285188 (+ 3015us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:35.287486 (+ 2298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.287493 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.287655 (+ 162us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.288339 (+ 684us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.289067 (+ 728us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.289074 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.289079 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.289247 (+ 168us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.289908 (+ 661us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:35.289914 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:35.290300 (+ 386us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.290507 (+ 207us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.290741 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":147,"threads_started":1}
I20260430 20:01:35.292770 12051 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41819
I20260430 20:01:35.292959 12051 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:35.293552 12051 heartbeater.cc:507] Master 127.10.46.62:41819 requested a full tablet report, sending...
I20260430 20:01:35.294600 11561 ts_manager.cc:194] Registered new tserver with Master: 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839)
I20260430 20:01:35.295285 11561 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:49955
I20260430 20:01:35.303676 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.295285 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:35421 (local address 127.10.46.62:41819)
0430 20:01:35.295417 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.295424 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.295601 (+ 177us) server_negotiation.cc:408] Connection header received
0430 20:01:35.295758 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.295762 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.295847 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.295932 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.297299 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.297805 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.298551 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.298691 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.299613 (+ 922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.299628 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.299631 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.299654 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.301510 (+ 1856us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.302043 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.302050 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.302052 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.302151 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.302574 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.302581 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.302586 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.302817 (+ 231us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.302945 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.303277 (+ 332us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.303406 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:01:35.303700 12057 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.295182 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.10.46.3:35421)
0430 20:01:35.295506 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.295524 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.295664 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.296140 (+ 476us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.296146 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.296677 (+ 531us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:35.297037 (+ 360us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.297049 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.297930 (+ 881us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.297934 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.298427 (+ 493us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.298435 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.298537 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.299132 (+ 595us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:35.299146 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:35.299500 (+ 354us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:35.301638 (+ 2138us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.301643 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.301645 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.301858 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.302297 (+ 439us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.302303 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.302307 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.302374 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.302943 (+ 569us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:35.302948 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:35.303062 (+ 114us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.303197 (+ 135us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.303407 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":107,"threads_started":1}
I20260430 20:01:35.303996 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:35.331940 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.322761 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.10.46.62:41819)
0430 20:01:35.322951 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.322955 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.322967 (+ 12us) server_negotiation.cc:408] Connection header received
0430 20:01:35.323044 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.323047 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.323092 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.323185 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.324084 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.324593 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.325241 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.325438 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.328285 (+ 2847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.328310 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.328312 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.328344 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.330246 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.330723 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.330727 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.330728 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.330780 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.331022 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.331026 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.331028 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.331202 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.331351 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.331594 (+ 243us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.331715 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260430 20:01:35.336246 11561 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:01:35.338728 11561 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:35.354763 12068 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.349829 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:40447 (local address 127.0.0.1:39012)
0430 20:01:35.350225 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.350272 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.350437 (+ 165us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.351447 (+ 1010us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.351453 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.351472 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.351843 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.351855 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.353193 (+ 1338us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.353197 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.354196 (+ 999us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.354206 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.354323 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.354390 (+ 67us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.354486 (+ 96us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.354564 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":120,"threads_started":1}
I20260430 20:01:35.354763 12070 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.350152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:32839 (local address 127.0.0.1:51092)
0430 20:01:35.350682 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.350695 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.350775 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.351706 (+ 931us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.351709 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.351731 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.352017 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.352028 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.353126 (+ 1098us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.353130 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.354196 (+ 1066us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.354206 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.354322 (+ 116us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.354390 (+ 68us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.354486 (+ 96us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.354564 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":136,"threads_started":1}
I20260430 20:01:35.355332 12069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.350057 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.10.46.2:40447)
0430 20:01:35.350943 (+ 886us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.350950 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.350973 (+ 23us) server_negotiation.cc:408] Connection header received
0430 20:01:35.351071 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.351079 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.351279 (+ 200us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.351433 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.351979 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.353055 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.354348 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.354898 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.355019 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.355083 (+ 64us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.355169 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":123,"threads_started":1}
I20260430 20:01:35.355679 12072 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.350577 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:46593 (local address 127.0.0.1:41624)
0430 20:01:35.351231 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.351244 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.351457 (+ 213us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.352057 (+ 600us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.352060 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.352076 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.352347 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.352354 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.353633 (+ 1279us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.353637 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.354536 (+ 899us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.354546 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.354651 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.354665 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.354714 (+ 49us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.354770 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":88,"threads_started":1}
I20260430 20:01:35.356194 12071 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.350539 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.10.46.3:32839)
0430 20:01:35.351235 (+ 696us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.351240 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.351287 (+ 47us) server_negotiation.cc:408] Connection header received
0430 20:01:35.351369 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.351376 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.351523 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.351729 (+ 206us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.352153 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.352988 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.355191 (+ 2203us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.355696 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.355841 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.355908 (+ 67us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.356000 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":99,"threads_started":1}
I20260430 20:01:35.357272 12073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.350754 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.10.46.1:46593)
0430 20:01:35.351054 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.351061 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.351392 (+ 331us) server_negotiation.cc:408] Connection header received
0430 20:01:35.351611 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.351617 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.351763 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.351917 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.352479 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.353497 (+ 1018us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.354858 (+ 1361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.356665 (+ 1807us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.356935 (+ 270us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.357022 (+ 87us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.357136 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260430 20:01:35.359102 11985 tablet_service.cc:1511] Processing CreateTablet for tablet 70ba910ce3774ab799fa9573ed314e7d (DEFAULT_TABLE table=test-table [id=568b118bf27345e2af1d9c58865150b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.359102 11843 tablet_service.cc:1511] Processing CreateTablet for tablet 70ba910ce3774ab799fa9573ed314e7d (DEFAULT_TABLE table=test-table [id=568b118bf27345e2af1d9c58865150b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.359648 11701 tablet_service.cc:1511] Processing CreateTablet for tablet 70ba910ce3774ab799fa9573ed314e7d (DEFAULT_TABLE table=test-table [id=568b118bf27345e2af1d9c58865150b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.360240 11843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba910ce3774ab799fa9573ed314e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.360232 11985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba910ce3774ab799fa9573ed314e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.360630 11701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba910ce3774ab799fa9573ed314e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.367118 12076 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Bootstrap starting.
I20260430 20:01:35.367444 12074 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Bootstrap starting.
I20260430 20:01:35.367919 12075 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Bootstrap starting.
I20260430 20:01:35.369735 12075 tablet_bootstrap.cc:654] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.369736 12076 tablet_bootstrap.cc:654] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.369736 12074 tablet_bootstrap.cc:654] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.370791 12074 log.cc:826] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:35.370846 12075 log.cc:826] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:35.371057 12076 log.cc:826] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:35.372877 12074 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: No bootstrap required, opened a new log
I20260430 20:01:35.372874 12075 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: No bootstrap required, opened a new log
I20260430 20:01:35.372874 12076 tablet_bootstrap.cc:492] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: No bootstrap required, opened a new log
I20260430 20:01:35.373101 12075 ts_tablet_manager.cc:1403] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:01:35.373101 12076 ts_tablet_manager.cc:1403] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 20:01:35.373101 12074 ts_tablet_manager.cc:1403] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260430 20:01:35.375689 12074 raft_consensus.cc:359] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.376145 12074 raft_consensus.cc:385] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.376241 12074 raft_consensus.cc:740] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730c5d154b4d1e82b2e59cc181c640, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.376343 12076 raft_consensus.cc:359] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.376343 12075 raft_consensus.cc:359] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.376554 12075 raft_consensus.cc:385] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.376602 12076 raft_consensus.cc:385] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.376649 12075 raft_consensus.cc:740] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447a1e8db614370b857a74613dd2340, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.376695 12076 raft_consensus.cc:740] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5182df9b9c1d4bc291ec250345706a6a, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.376742 12074 consensus_queue.cc:260] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.377094 12075 consensus_queue.cc:260] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.377094 12076 consensus_queue.cc:260] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.377831 12051 heartbeater.cc:499] Master 127.10.46.62:41819 was elected leader, sending a full tablet report...
I20260430 20:01:35.377981 12076 ts_tablet_manager.cc:1434] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:01:35.378026 12074 ts_tablet_manager.cc:1434] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260430 20:01:35.378049 11909 heartbeater.cc:499] Master 127.10.46.62:41819 was elected leader, sending a full tablet report...
I20260430 20:01:35.378372 11767 heartbeater.cc:499] Master 127.10.46.62:41819 was elected leader, sending a full tablet report...
I20260430 20:01:35.379462 12075 ts_tablet_manager.cc:1434] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260430 20:01:35.412554 12082 raft_consensus.cc:493] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:35.412755 12082 raft_consensus.cc:515] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.414659 12082 leader_election.cc:290] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839)
I20260430 20:01:35.417908 11802 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.414873 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:32839 (local address 127.10.46.2:47819)
0430 20:01:35.415059 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.415079 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.415306 (+ 227us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.415651 (+ 345us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.415654 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.415677 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.416000 (+ 323us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.416007 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.416747 (+ 740us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.416753 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.417446 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.417456 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.417616 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.417636 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.417693 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.417757 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260430 20:01:35.418347 12071 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.415087 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:47819 (local address 127.10.46.3:32839)
0430 20:01:35.415396 (+ 309us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.415403 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.415421 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:01:35.415479 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.415482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.415543 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.415626 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.416117 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.416633 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.417591 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.418075 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.418119 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.418178 (+ 59us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.418261 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260430 20:01:35.418932 12005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ba910ce3774ab799fa9573ed314e7d" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447a1e8db614370b857a74613dd2340" is_pre_election: true
I20260430 20:01:35.419063 12083 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.414873 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:46593 (local address 127.10.46.2:51777)
0430 20:01:35.415672 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.415688 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.416030 (+ 342us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.416330 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.416333 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.416351 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.416599 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.416604 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.417879 (+ 1275us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.417886 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.418665 (+ 779us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.418674 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.418804 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.418818 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.418871 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.418923 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":357,"thread_start_us":138,"threads_started":1}
I20260430 20:01:35.419271 12005 raft_consensus.cc:2468] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 0.
I20260430 20:01:35.419565 12073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.415077 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:51777 (local address 127.10.46.1:46593)
0430 20:01:35.415220 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.415224 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.415954 (+ 730us) server_negotiation.cc:408] Connection header received
0430 20:01:35.416166 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.416169 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.416213 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.416298 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.416715 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.417740 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.418789 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.419308 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.419340 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.419417 (+ 77us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.419473 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 20:01:35.419837 11796 leader_election.cc:304] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447a1e8db614370b857a74613dd2340, 5182df9b9c1d4bc291ec250345706a6a; no voters:
I20260430 20:01:35.420109 12082 raft_consensus.cc:2804] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:35.420197 12082 raft_consensus.cc:493] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:35.420186 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ba910ce3774ab799fa9573ed314e7d" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640" is_pre_election: true
I20260430 20:01:35.420259 12082 raft_consensus.cc:3060] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.420433 11721 raft_consensus.cc:2468] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 0.
I20260430 20:01:35.421499 12082 raft_consensus.cc:515] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.421916 12082 leader_election.cc:290] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 election: Requested vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839)
I20260430 20:01:35.422633 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ba910ce3774ab799fa9573ed314e7d" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640"
I20260430 20:01:35.422626 12005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ba910ce3774ab799fa9573ed314e7d" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447a1e8db614370b857a74613dd2340"
I20260430 20:01:35.422777 11721 raft_consensus.cc:3060] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.422809 12005 raft_consensus.cc:3060] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.423929 11721 raft_consensus.cc:2468] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 1.
I20260430 20:01:35.424221 12005 raft_consensus.cc:2468] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 1.
I20260430 20:01:35.424438 11794 leader_election.cc:304] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d730c5d154b4d1e82b2e59cc181c640, 5182df9b9c1d4bc291ec250345706a6a; no voters:
I20260430 20:01:35.424718 12082 raft_consensus.cc:2804] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:35.425024 12082 raft_consensus.cc:697] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 1 LEADER]: Becoming Leader. State: Replica: 5182df9b9c1d4bc291ec250345706a6a, State: Running, Role: LEADER
I20260430 20:01:35.425452 12082 consensus_queue.cc:237] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.429854 11560 catalog_manager.cc:5671] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2). New cstate: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:35.464838 12069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.461365 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.10.46.2:40447)
0430 20:01:35.461516 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.461520 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.461533 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:01:35.461608 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.461611 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.461666 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.461755 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:35.462409 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.462943 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.463708 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.463890 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.463941 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.464369 (+ 428us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.464465 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.464665 (+ 200us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.464718 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 20:01:35.471024 11721 raft_consensus.cc:1275] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Refusing update from remote peer 5182df9b9c1d4bc291ec250345706a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.471024 12005 raft_consensus.cc:1275] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Refusing update from remote peer 5182df9b9c1d4bc291ec250345706a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.472045 12084 consensus_queue.cc:1048] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:35.472424 12082 consensus_queue.cc:1048] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:35.491524 11561 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:35.491695 11561 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:35.494791 11561 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Sending DeleteTablet for 3 replicas of tablet 70ba910ce3774ab799fa9573ed314e7d
I20260430 20:01:35.496181 11843 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ba910ce3774ab799fa9573ed314e7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:35 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260430 20:01:35.496253 11701 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ba910ce3774ab799fa9573ed314e7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:35 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:41624
I20260430 20:01:35.496784 12098 tablet_replica.cc:333] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: stopping tablet replica
I20260430 20:01:35.497160 11985 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ba910ce3774ab799fa9573ed314e7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:35 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:51092
I20260430 20:01:35.497334 12098 raft_consensus.cc:2243] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:35.497717 12098 raft_consensus.cc:2272] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:35.497756 12099 tablet_replica.cc:333] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: stopping tablet replica
I20260430 20:01:35.497737 11561 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 20:01:35.498011 12099 raft_consensus.cc:2243] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260430 20:01:35.498260 11561 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:35.498533 12099 raft_consensus.cc:2272] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:35.499173 12097 tablet_replica.cc:333] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: stopping tablet replica
I20260430 20:01:35.499477 12097 raft_consensus.cc:2243] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:01:35.500061 12097 raft_consensus.cc:2272] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:35.500267 12099 ts_tablet_manager.cc:1916] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:35.502847 12098 ts_tablet_manager.cc:1916] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:35.503175 12099 ts_tablet_manager.cc:1929] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:01:35.503357 12099 log.cc:1199] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/70ba910ce3774ab799fa9573ed314e7d
I20260430 20:01:35.503741 12099 ts_tablet_manager.cc:1950] T 70ba910ce3774ab799fa9573ed314e7d P 1447a1e8db614370b857a74613dd2340: Deleting consensus metadata
I20260430 20:01:35.504580 11843 tablet_service.cc:1511] Processing CreateTablet for tablet 12427f3357f547788e211e0dbc716690 (DEFAULT_TABLE table=test-table [id=ec50b609f4be49ecb6f7065d67f561cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.504863 11843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12427f3357f547788e211e0dbc716690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.504845 11701 tablet_service.cc:1511] Processing CreateTablet for tablet 12427f3357f547788e211e0dbc716690 (DEFAULT_TABLE table=test-table [id=ec50b609f4be49ecb6f7065d67f561cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.505093 11701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12427f3357f547788e211e0dbc716690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.507442 12076 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Bootstrap starting.
I20260430 20:01:35.507519 12098 ts_tablet_manager.cc:1929] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:01:35.507604 12098 log.cc:1199] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/70ba910ce3774ab799fa9573ed314e7d
I20260430 20:01:35.507786 12074 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Bootstrap starting.
I20260430 20:01:35.507929 12098 ts_tablet_manager.cc:1950] T 70ba910ce3774ab799fa9573ed314e7d P 3d730c5d154b4d1e82b2e59cc181c640: Deleting consensus metadata
I20260430 20:01:35.508307 11985 tablet_service.cc:1511] Processing CreateTablet for tablet 12427f3357f547788e211e0dbc716690 (DEFAULT_TABLE table=test-table [id=ec50b609f4be49ecb6f7065d67f561cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:35.508572 11985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12427f3357f547788e211e0dbc716690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.508817 12074 tablet_bootstrap.cc:654] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.508922 12097 ts_tablet_manager.cc:1916] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:35.510701 11546 catalog_manager.cc:5002] TS 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593): tablet 70ba910ce3774ab799fa9573ed314e7d (table test-table [id=568b118bf27345e2af1d9c58865150b8]) successfully deleted
I20260430 20:01:35.510851 12075 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Bootstrap starting.
I20260430 20:01:35.512009 12076 tablet_bootstrap.cc:654] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.512055 12075 tablet_bootstrap.cc:654] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.513255 12097 ts_tablet_manager.cc:1929] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:01:35.513409 12076 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: No bootstrap required, opened a new log
I20260430 20:01:35.513532 12076 ts_tablet_manager.cc:1403] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260430 20:01:35.514048 12076 raft_consensus.cc:359] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.513411 12097 log.cc:1199] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/70ba910ce3774ab799fa9573ed314e7d
I20260430 20:01:35.514175 12076 raft_consensus.cc:385] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.514212 12076 raft_consensus.cc:740] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5182df9b9c1d4bc291ec250345706a6a, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.514220 12074 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: No bootstrap required, opened a new log
I20260430 20:01:35.514315 12074 ts_tablet_manager.cc:1403] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260430 20:01:35.514349 12076 consensus_queue.cc:260] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.514488 12097 ts_tablet_manager.cc:1950] T 70ba910ce3774ab799fa9573ed314e7d P 5182df9b9c1d4bc291ec250345706a6a: Deleting consensus metadata
I20260430 20:01:35.514605 12076 ts_tablet_manager.cc:1434] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:35.514803 11548 catalog_manager.cc:5002] TS 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839): tablet 70ba910ce3774ab799fa9573ed314e7d (table test-table [id=568b118bf27345e2af1d9c58865150b8]) successfully deleted
I20260430 20:01:35.514833 12074 raft_consensus.cc:359] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.514928 12074 raft_consensus.cc:385] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.514957 12074 raft_consensus.cc:740] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730c5d154b4d1e82b2e59cc181c640, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.515092 12074 consensus_queue.cc:260] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.515342 12074 ts_tablet_manager.cc:1434] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260430 20:01:35.516232 12075 tablet_bootstrap.cc:492] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: No bootstrap required, opened a new log
I20260430 20:01:35.516309 11546 catalog_manager.cc:5002] TS 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2:40447): tablet 70ba910ce3774ab799fa9573ed314e7d (table test-table [id=568b118bf27345e2af1d9c58865150b8]) successfully deleted
I20260430 20:01:35.516341 12075 ts_tablet_manager.cc:1403] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260430 20:01:35.516887 12075 raft_consensus.cc:359] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.516984 12075 raft_consensus.cc:385] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.517016 12075 raft_consensus.cc:740] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447a1e8db614370b857a74613dd2340, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.517140 12075 consensus_queue.cc:260] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.517383 12075 ts_tablet_manager.cc:1434] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260430 20:01:35.526950 11768 tablet.cc:2404] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:01:35.539633 12052 tablet.cc:2404] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:01:35.540367 11910 tablet.cc:2404] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:01:35.594815 12081 raft_consensus.cc:493] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:35.595088 12081 raft_consensus.cc:515] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.597258 12081 leader_election.cc:290] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2:40447)
I20260430 20:01:35.603718 12069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.598826 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:57975 (local address 127.10.46.2:40447)
0430 20:01:35.599048 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.599054 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.599068 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:01:35.599114 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.599140 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.599190 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.599279 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.600715 (+ 1436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.601509 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.602744 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.603241 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.603276 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.603529 (+ 253us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.603588 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260430 20:01:35.603868 12073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.597678 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:59479 (local address 127.10.46.1:46593)
0430 20:01:35.597812 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.597817 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.597926 (+ 109us) server_negotiation.cc:408] Connection header received
0430 20:01:35.598116 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.598119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.598213 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.598315 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:35.599693 (+ 1378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.600567 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.602178 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.602861 (+ 683us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.603022 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.603222 (+ 200us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.603525 (+ 303us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 20:01:35.603827 11944 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.597639 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:46593 (local address 127.10.46.3:59479)
0430 20:01:35.597827 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.597843 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.597991 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.598373 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.598376 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.598395 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.599555 (+ 1160us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.599563 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.600711 (+ 1148us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.600715 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.602014 (+ 1299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.602022 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.602952 (+ 930us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.602966 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.603254 (+ 288us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.603439 (+ 185us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260430 20:01:35.604646 12103 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.598351 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:40447 (local address 127.10.46.3:57975)
0430 20:01:35.598826 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.598840 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.598922 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.599290 (+ 368us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.599293 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.599311 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:35.599714 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.599720 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.601705 (+ 1985us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.601710 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.602602 (+ 892us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.602610 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.603385 (+ 775us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.603397 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.604450 (+ 1053us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.604511 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":119,"threads_started":1}
I20260430 20:01:35.604737 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12427f3357f547788e211e0dbc716690" candidate_uuid: "1447a1e8db614370b857a74613dd2340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640" is_pre_election: true
I20260430 20:01:35.605129 11721 raft_consensus.cc:2468] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1447a1e8db614370b857a74613dd2340 in term 0.
I20260430 20:01:35.605422 11863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12427f3357f547788e211e0dbc716690" candidate_uuid: "1447a1e8db614370b857a74613dd2340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5182df9b9c1d4bc291ec250345706a6a" is_pre_election: true
I20260430 20:01:35.605582 11936 leader_election.cc:304] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447a1e8db614370b857a74613dd2340, 3d730c5d154b4d1e82b2e59cc181c640; no voters:
I20260430 20:01:35.605638 11863 raft_consensus.cc:2468] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1447a1e8db614370b857a74613dd2340 in term 0.
I20260430 20:01:35.605825 12081 raft_consensus.cc:2804] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:35.605885 12081 raft_consensus.cc:493] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:35.605926 12081 raft_consensus.cc:3060] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.606976 12081 raft_consensus.cc:515] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.607362 12081 leader_election.cc:290] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [CANDIDATE]: Term 1 election: Requested vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2:40447)
I20260430 20:01:35.607724 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12427f3357f547788e211e0dbc716690" candidate_uuid: "1447a1e8db614370b857a74613dd2340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640"
I20260430 20:01:35.607836 11863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12427f3357f547788e211e0dbc716690" candidate_uuid: "1447a1e8db614370b857a74613dd2340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5182df9b9c1d4bc291ec250345706a6a"
I20260430 20:01:35.607913 11721 raft_consensus.cc:3060] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.607961 11863 raft_consensus.cc:3060] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.608800 11721 raft_consensus.cc:2468] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1447a1e8db614370b857a74613dd2340 in term 1.
I20260430 20:01:35.609031 11863 raft_consensus.cc:2468] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1447a1e8db614370b857a74613dd2340 in term 1.
I20260430 20:01:35.609117 11936 leader_election.cc:304] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447a1e8db614370b857a74613dd2340, 3d730c5d154b4d1e82b2e59cc181c640; no voters:
I20260430 20:01:35.609375 12081 raft_consensus.cc:2804] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:35.609623 12081 raft_consensus.cc:697] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 1 LEADER]: Becoming Leader. State: Replica: 1447a1e8db614370b857a74613dd2340, State: Running, Role: LEADER
I20260430 20:01:35.609879 12081 consensus_queue.cc:237] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.612602 11561 catalog_manager.cc:5671] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1447a1e8db614370b857a74613dd2340 (127.10.46.3). New cstate: current_term: 1 leader_uuid: "1447a1e8db614370b857a74613dd2340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } health_report { overall_health: HEALTHY } } }
Apr 30 20:01:35 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:35.662241 12111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.650415 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41819 (local address 127.0.0.1:46762)
0430 20:01:35.650737 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.650769 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.650965 (+ 196us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.651301 (+ 336us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.651304 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.652178 (+ 874us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:35.652654 (+ 476us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.652668 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.653873 (+ 1205us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.653877 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.654304 (+ 427us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.654312 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.654426 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.655039 (+ 613us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:35.655052 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:35.656712 (+ 1660us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:35.659859 (+ 3147us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.659872 (+ 13us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.659877 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.660174 (+ 297us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.660707 (+ 533us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:35.660714 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:35.660720 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:35.660806 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:35.661330 (+ 524us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:35.661339 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:35.661467 (+ 128us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.661600 (+ 133us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.661979 (+ 379us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260430 20:01:35.662240 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.650538 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.10.46.62:41819)
0430 20:01:35.650702 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.650707 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.650894 (+ 187us) server_negotiation.cc:408] Connection header received
0430 20:01:35.651090 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.651094 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.651146 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.651246 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:35.652880 (+ 1634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.653685 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.654443 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.654639 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.656862 (+ 2223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:35.656886 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:35.656890 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:35.656920 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:35.659186 (+ 2266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.660351 (+ 1165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.660357 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.660359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.660459 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:35.660923 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:35.660927 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:35.660929 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:35.661170 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:35.661303 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.661672 (+ 369us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.662074 (+ 402us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":251,"server-negotiator.queue_time_us":61}
I20260430 20:01:35.666723 11561 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:46762:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 20:01:35.668021 11561 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:35.675115 11701 tablet_service.cc:1511] Processing CreateTablet for tablet 3ff9498a47f34489b65e8be7a9576aff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a1ccb656ffc47bfbf3d4b7d39f19e18]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:35.675256 11843 tablet_service.cc:1511] Processing CreateTablet for tablet 3ff9498a47f34489b65e8be7a9576aff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a1ccb656ffc47bfbf3d4b7d39f19e18]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:35.675565 11701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ff9498a47f34489b65e8be7a9576aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.675622 11843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ff9498a47f34489b65e8be7a9576aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.675513 11985 tablet_service.cc:1511] Processing CreateTablet for tablet 3ff9498a47f34489b65e8be7a9576aff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a1ccb656ffc47bfbf3d4b7d39f19e18]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:35.675910 11985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ff9498a47f34489b65e8be7a9576aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:35.678531 12074 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: Bootstrap starting.
I20260430 20:01:35.678531 12076 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: Bootstrap starting.
I20260430 20:01:35.680037 12074 tablet_bootstrap.cc:654] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.680037 12076 tablet_bootstrap.cc:654] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.680620 12075 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: Bootstrap starting.
I20260430 20:01:35.681439 12074 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: No bootstrap required, opened a new log
I20260430 20:01:35.681555 12074 ts_tablet_manager.cc:1403] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:35.681713 12075 tablet_bootstrap.cc:654] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:35.682163 12074 raft_consensus.cc:359] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.682302 12074 raft_consensus.cc:385] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.682348 12074 raft_consensus.cc:740] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730c5d154b4d1e82b2e59cc181c640, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.682482 12074 consensus_queue.cc:260] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.682754 12074 ts_tablet_manager.cc:1434] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:35.683058 12075 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: No bootstrap required, opened a new log
I20260430 20:01:35.683024 12080 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.683131 12075 ts_tablet_manager.cc:1403] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:35.683151 12080 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.683557 12075 raft_consensus.cc:359] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.683657 12075 raft_consensus.cc:385] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.683686 12075 raft_consensus.cc:740] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447a1e8db614370b857a74613dd2340, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.683842 12075 consensus_queue.cc:260] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.684079 12076 tablet_bootstrap.cc:492] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: No bootstrap required, opened a new log
I20260430 20:01:35.684057 12081 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.684144 12081 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.684167 12076 ts_tablet_manager.cc:1403] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260430 20:01:35.684353 12075 ts_tablet_manager.cc:1434] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:35.685910 12076 raft_consensus.cc:359] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.686079 12076 raft_consensus.cc:385] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:35.686126 12076 raft_consensus.cc:740] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5182df9b9c1d4bc291ec250345706a6a, State: Initialized, Role: FOLLOWER
I20260430 20:01:35.686501 12076 consensus_queue.cc:260] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.686913 12076 ts_tablet_manager.cc:1434] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260430 20:01:35.686861 12094 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.687042 12094 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.908290 12094 raft_consensus.cc:493] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:35.908622 12094 raft_consensus.cc:515] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.909368 12094 leader_election.cc:290] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839)
I20260430 20:01:35.909873 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ff9498a47f34489b65e8be7a9576aff" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640" is_pre_election: true
I20260430 20:01:35.909876 12005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ff9498a47f34489b65e8be7a9576aff" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447a1e8db614370b857a74613dd2340" is_pre_election: true
I20260430 20:01:35.910037 11721 raft_consensus.cc:2468] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 0.
I20260430 20:01:35.910049 12005 raft_consensus.cc:2468] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 0.
I20260430 20:01:35.910392 11794 leader_election.cc:304] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d730c5d154b4d1e82b2e59cc181c640, 5182df9b9c1d4bc291ec250345706a6a; no voters:
I20260430 20:01:35.910564 12094 raft_consensus.cc:2804] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:35.910625 12094 raft_consensus.cc:493] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:35.910686 12094 raft_consensus.cc:3060] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.911842 12094 raft_consensus.cc:515] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.912293 12094 leader_election.cc:290] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 election: Requested vote from peers 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593), 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839)
I20260430 20:01:35.912721 11721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ff9498a47f34489b65e8be7a9576aff" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730c5d154b4d1e82b2e59cc181c640"
I20260430 20:01:35.912878 12005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ff9498a47f34489b65e8be7a9576aff" candidate_uuid: "5182df9b9c1d4bc291ec250345706a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447a1e8db614370b857a74613dd2340"
I20260430 20:01:35.913048 11721 raft_consensus.cc:3060] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.913225 12005 raft_consensus.cc:3060] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:35.914345 12005 raft_consensus.cc:2468] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 1.
I20260430 20:01:35.914345 11721 raft_consensus.cc:2468] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5182df9b9c1d4bc291ec250345706a6a in term 1.
I20260430 20:01:35.914698 11794 leader_election.cc:304] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d730c5d154b4d1e82b2e59cc181c640, 5182df9b9c1d4bc291ec250345706a6a; no voters:
I20260430 20:01:35.914919 12094 raft_consensus.cc:2804] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:35.915038 12094 raft_consensus.cc:697] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [term 1 LEADER]: Becoming Leader. State: Replica: 5182df9b9c1d4bc291ec250345706a6a, State: Running, Role: LEADER
I20260430 20:01:35.915187 12094 consensus_queue.cc:237] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } }
I20260430 20:01:35.916349 12084 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: TxnStatusTablet state changed. Reason: New leader 5182df9b9c1d4bc291ec250345706a6a. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.916466 12084 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:35.916702 12116 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 20:01:35.916886 12116 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 20:01:35.917692 11561 catalog_manager.cc:5671] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2). New cstate: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:35.934965 12069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.931681 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.10.46.2:40447)
0430 20:01:35.931862 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.931867 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.931882 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:01:35.932245 (+ 363us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.932248 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.932306 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.932386 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:35.932820 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.933380 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.934336 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.934495 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.934533 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.934614 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.934682 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.934814 (+ 132us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.934849 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:01:35.935012 12111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.931550 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:40447 (local address 127.0.0.1:39030)
0430 20:01:35.931798 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.931817 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.931952 (+ 135us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.932421 (+ 469us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.932425 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.932435 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:35.932702 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.932708 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.933509 (+ 801us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.933513 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.934171 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.934179 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.934311 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.934362 (+ 51us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.934709 (+ 347us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.934719 (+ 10us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.934795 (+ 76us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.934867 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260430 20:01:35.938582 12005 raft_consensus.cc:1275] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Refusing update from remote peer 5182df9b9c1d4bc291ec250345706a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.938582 11721 raft_consensus.cc:1275] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Refusing update from remote peer 5182df9b9c1d4bc291ec250345706a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.939069 12084 consensus_queue.cc:1048] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:35.939286 12094 consensus_queue.cc:1048] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:35.941188 12081 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: TxnStatusTablet state changed. Reason: New leader 5182df9b9c1d4bc291ec250345706a6a. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.941409 12081 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.941951 12080 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: TxnStatusTablet state changed. Reason: New leader 5182df9b9c1d4bc291ec250345706a6a. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.942070 12080 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.944167 12082 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.944291 12082 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:35.944876 12087 mvcc.cc:204] Tried to move back new op lower bound from 7280964796159569920 to 7280964796071075840. Current Snapshot: MvccSnapshot[applied={T|T < 7280964796159569920 or (T in {7280964796159569920})}]
I20260430 20:01:35.945175 12081 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.945287 12081 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 1447a1e8db614370b857a74613dd2340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.945369 12088 mvcc.cc:204] Tried to move back new op lower bound from 7280964796159569920 to 7280964796071075840. Current Snapshot: MvccSnapshot[applied={T|T < 7280964796159569920 or (T in {7280964796159569920})}]
I20260430 20:01:35.947746 12094 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.947889 12094 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 5182df9b9c1d4bc291ec250345706a6a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:35.948593 12080 tablet_replica.cc:442] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5182df9b9c1d4bc291ec250345706a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 } } peers { permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 } } peers { permanent_uuid: "1447a1e8db614370b857a74613dd2340" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 32839 } } }
I20260430 20:01:35.948599 12089 mvcc.cc:204] Tried to move back new op lower bound from 7280964796159569920 to 7280964796071075840. Current Snapshot: MvccSnapshot[applied={T|T < 7280964796159569920}]
I20260430 20:01:35.948731 12080 tablet_replica.cc:445] T 3ff9498a47f34489b65e8be7a9576aff P 3d730c5d154b4d1e82b2e59cc181c640: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:35.956275 12121 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.951995 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.10.46.3:32839)
0430 20:01:35.952412 (+ 417us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.952417 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.952595 (+ 178us) server_negotiation.cc:408] Connection header received
0430 20:01:35.952816 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.952819 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.952898 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.953020 (+ 122us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:35.953545 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.954107 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.955016 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.955162 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.955217 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.955725 (+ 508us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.955893 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.956009 (+ 116us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.956061 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260430 20:01:35.970482 12124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.965987 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:40447 (local address 127.10.46.3:41363)
0430 20:01:35.966542 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:35.966557 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:35.966703 (+ 146us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:35.967138 (+ 435us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:35.967142 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:35.967152 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:35.967455 (+ 303us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.967460 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.968600 (+ 1140us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.968605 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:35.969282 (+ 677us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:35.969298 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.969447 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.969495 (+ 48us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.969923 (+ 428us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.969930 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:01:35.970042 (+ 112us) client_negotiation.cc:241] Negotiation successful
0430 20:01:35.970118 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":196,"threads_started":1}
I20260430 20:01:35.970530 12069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:35.966148 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:41363 (local address 127.10.46.2:40447)
0430 20:01:35.966341 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:35.966346 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:35.966657 (+ 311us) server_negotiation.cc:408] Connection header received
0430 20:01:35.966823 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:35.966827 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:35.966899 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:35.967029 (+ 130us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:35.967595 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.968402 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:35.969446 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:35.969650 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:35.969695 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:35.969795 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:35.969887 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:35.970059 (+ 172us) server_negotiation.cc:300] Negotiation successful
0430 20:01:35.970107 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260430 20:01:35.980737 11721 raft_consensus.cc:1275] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Refusing update from remote peer 1447a1e8db614370b857a74613dd2340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.980953 11863 raft_consensus.cc:1275] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Refusing update from remote peer 1447a1e8db614370b857a74613dd2340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:35.981386 12104 consensus_queue.cc:1048] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d730c5d154b4d1e82b2e59cc181c640" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:35.981611 12081 consensus_queue.cc:1048] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5182df9b9c1d4bc291ec250345706a6a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:36.012969 12121 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:36.010074 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:57421 (local address 127.10.46.3:32839)
0430 20:01:36.010207 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:36.010211 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:36.010227 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:01:36.010287 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:36.010289 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:36.010339 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:36.010426 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:36.010833 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.011355 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:36.012170 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.012387 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:36.012428 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:36.012535 (+ 107us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:36.012628 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:36.012766 (+ 138us) server_negotiation.cc:300] Negotiation successful
0430 20:01:36.012822 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260430 20:01:36.012988 12129 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:36.009692 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:32839 (local address 127.10.46.2:57421)
0430 20:01:36.010023 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:36.010041 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:36.010151 (+ 110us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:36.010446 (+ 295us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:36.010449 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:36.010458 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:36.010714 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:36.010721 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.011468 (+ 747us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:36.011474 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:36.012009 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:36.012018 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.012175 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:36.012223 (+ 48us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:36.012655 (+ 432us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:36.012662 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:01:36.012740 (+ 78us) client_negotiation.cc:241] Negotiation successful
0430 20:01:36.012816 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260430 20:01:36.053689 11560 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:36.053877 11560 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:36.056633 11560 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 29e06c6e82374a46a3f524d95865a257: Sending DeleteTablet for 3 replicas of tablet 12427f3357f547788e211e0dbc716690
I20260430 20:01:36.057492 11985 tablet_service.cc:1558] Processing DeleteTablet for tablet 12427f3357f547788e211e0dbc716690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:36 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:51092
I20260430 20:01:36.058490 11701 tablet_service.cc:1558] Processing DeleteTablet for tablet 12427f3357f547788e211e0dbc716690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:36 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:41624
I20260430 20:01:36.058658 11843 tablet_service.cc:1558] Processing DeleteTablet for tablet 12427f3357f547788e211e0dbc716690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:36 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260430 20:01:36.064025 12132 tablet_replica.cc:333] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: stopping tablet replica
I20260430 20:01:36.064049 12133 tablet_replica.cc:333] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: stopping tablet replica
I20260430 20:01:36.064208 12133 raft_consensus.cc:2243] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:36.064276 12132 raft_consensus.cc:2243] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:01:36.064492 12133 raft_consensus.cc:2272] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:36.064713 12132 raft_consensus.cc:2272] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:36.064826 12134 tablet_replica.cc:333] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: stopping tablet replica
I20260430 20:01:36.065006 12134 raft_consensus.cc:2243] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:36.065179 12134 raft_consensus.cc:2272] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:36.065464 12132 ts_tablet_manager.cc:1916] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:36.065464 12133 ts_tablet_manager.cc:1916] T 12427f3357f547788e211e0dbc716690 P 3d730c5d154b4d1e82b2e59cc181c640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:36.066628 12134 ts_tablet_manager.cc:1916] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
Apr 30 20:01:36 dist-test-slave-0bdw krb5kdc[11514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:01:36.082965 12073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:36.065804 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.10.46.1:46593)
0430 20:01:36.065967 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:36.065974 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:36.071848 (+ 5874us) server_negotiation.cc:408] Connection header received
0430 20:01:36.072086 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:36.072090 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:36.072150 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:36.072206 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:36.073451 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.074193 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:36.075124 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.075300 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:36.078763 (+ 3463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:36.078806 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:36.078823 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:36.078862 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:36.081130 (+ 2268us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:36.081682 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:36.081689 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:36.081691 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:36.081750 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:36.081991 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:36.081995 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:36.081997 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:36.082225 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:36.082436 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:36.082638 (+ 202us) server_negotiation.cc:300] Negotiation successful
0430 20:01:36.082778 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260430 20:01:36.083459 11761 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41640
I20260430 20:01:36.095747 11611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:36.086138 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.10.46.62:41819)
0430 20:01:36.086279 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:36.086283 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:36.086648 (+ 365us) server_negotiation.cc:408] Connection header received
0430 20:01:36.086787 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:36.086792 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:36.086878 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:36.087002 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:36.088369 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.089179 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:36.090148 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:36.090712 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:36.091706 (+ 994us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:36.091735 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:36.091738 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:36.091789 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:36.093647 (+ 1858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:36.094375 (+ 728us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:36.094382 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:36.094384 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:36.094560 (+ 176us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:36.094831 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:36.094834 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:36.094836 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:36.095025 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:36.095191 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:36.095443 (+ 252us) server_negotiation.cc:300] Negotiation successful
0430 20:01:36.095568 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260430 20:01:36.096704 11560 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766
I20260430 20:01:36.098454 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11636
I20260430 20:01:36.123895 12134 ts_tablet_manager.cc:1929] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:36.124006 12134 log.cc:1199] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/12427f3357f547788e211e0dbc716690
I20260430 20:01:36.124174 12132 ts_tablet_manager.cc:1929] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:36.124267 12132 log.cc:1199] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/12427f3357f547788e211e0dbc716690
I20260430 20:01:36.124295 12134 ts_tablet_manager.cc:1950] T 12427f3357f547788e211e0dbc716690 P 5182df9b9c1d4bc291ec250345706a6a: Deleting consensus metadata
I20260430 20:01:36.124573 12132 ts_tablet_manager.cc:1950] T 12427f3357f547788e211e0dbc716690 P 1447a1e8db614370b857a74613dd2340: Deleting consensus metadata
I20260430 20:01:36.125514 11548 catalog_manager.cc:5002] TS 1447a1e8db614370b857a74613dd2340 (127.10.46.3:32839): tablet 12427f3357f547788e211e0dbc716690 (table test-table [id=ec50b609f4be49ecb6f7065d67f561cd]) successfully deleted
I20260430 20:01:36.125782 11546 catalog_manager.cc:5002] TS 5182df9b9c1d4bc291ec250345706a6a (127.10.46.2:40447): tablet 12427f3357f547788e211e0dbc716690 (table test-table [id=ec50b609f4be49ecb6f7065d67f561cd]) successfully deleted
W20260430 20:01:36.129637 11546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.46.1:46593 (error 108)
I20260430 20:01:36.131071 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11778
I20260430 20:01:36.131361 12140 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:36.130476 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:46593 (local address 127.0.0.1:41650)
0430 20:01:36.130927 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:36.131106 (+ 179us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":68,"threads_started":1}
W20260430 20:01:36.133384 11546 catalog_manager.cc:4729] TS 3d730c5d154b4d1e82b2e59cc181c640 (127.10.46.1:46593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12427f3357f547788e211e0dbc716690: Network error: Client connection negotiation failed: client connection to 127.10.46.1:46593: connect: Connection refused (error 111)
I20260430 20:01:36.145184 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11920
I20260430 20:01:36.156862 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 11530
2026-04-30T20:01:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4171 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:36 dist-test-slave-0bdw krb5kdc[12143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:36 dist-test-slave-0bdw krb5kdc[12143](info): set up 2 sockets
Apr 30 20:01:36 dist-test-slave-0bdw krb5kdc[12143](info): commencing operation
krb5kdc: starting...
W20260430 20:01:38.256392 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.075s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:38 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:38.417341 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35489
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:38.540540 12159 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:38.540915 12159 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:38.541021 12159 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:38.544829 12159 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:38.544943 12159 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 20:01:38.544977 12159 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:38.545029 12159 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:38.545085 12159 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:38.545128 12159 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:38.550776 12159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:35489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:38.552131 12159 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:38.553275 12159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:38.559371 12164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:38.559492 12167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:38.559680 12159 server_base.cc:1061] running on GCE node
W20260430 20:01:38.559492 12165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:38.560561 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:38.561930 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:38.563139 12159 hybrid_clock.cc:648] HybridClock initialized: now 1777579298563122 us; error 35 us; skew 500 ppm
Apr 30 20:01:38 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:38.566165 12159 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:38.567384 12159 webserver.cc:492] Webserver started at http://127.10.46.62:41349/ using document root <none> and password file <none>
I20260430 20:01:38.568085 12159 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:38.568146 12159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:38.568325 12159 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:38.570405 12159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "94f6b46ea13c4794aaccdf169d3737fc"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "f1c9b8bb63a9669d7b1e9445c587fb28"
server_key_iv: "4ab6a62832923e1b617dae092abb218d"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.570932 12159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "94f6b46ea13c4794aaccdf169d3737fc"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "f1c9b8bb63a9669d7b1e9445c587fb28"
server_key_iv: "4ab6a62832923e1b617dae092abb218d"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.575371 12159 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260430 20:01:38.578294 12174 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:38.579981 12159 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:38.580403 12159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "94f6b46ea13c4794aaccdf169d3737fc"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "f1c9b8bb63a9669d7b1e9445c587fb28"
server_key_iv: "4ab6a62832923e1b617dae092abb218d"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.580595 12159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:38.601433 12159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:38.604782 12159 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:38.605034 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:38.614589 12159 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:35489
I20260430 20:01:38.614626 12236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:35489 every 8 connection(s)
I20260430 20:01:38.615904 12159 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:38.619331 12237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:38.624586 12237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Bootstrap starting.
I20260430 20:01:38.625419 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12159
I20260430 20:01:38.625670 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:38.625942 10424 external_mini_cluster.cc:1468] Setting key dbe3929149834cb75134be6fefadd102
I20260430 20:01:38.627398 12237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:38.628249 12237 log.cc:826] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:38.630368 12237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: No bootstrap required, opened a new log
I20260430 20:01:38.633888 12237 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } }
I20260430 20:01:38.634312 12237 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:38.634459 12237 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f6b46ea13c4794aaccdf169d3737fc, State: Initialized, Role: FOLLOWER
I20260430 20:01:38.635083 12237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } }
I20260430 20:01:38.635347 12237 raft_consensus.cc:399] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:38.635453 12237 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:38.635583 12237 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:38.638185 12237 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } }
I20260430 20:01:38.638751 12237 leader_election.cc:304] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f6b46ea13c4794aaccdf169d3737fc; no voters:
I20260430 20:01:38.639202 12237 leader_election.cc:290] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:38.639336 12242 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:38.639606 12242 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [term 1 LEADER]: Becoming Leader. State: Replica: 94f6b46ea13c4794aaccdf169d3737fc, State: Running, Role: LEADER
I20260430 20:01:38.640051 12242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } }
I20260430 20:01:38.640631 12237 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:38.641827 12243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f6b46ea13c4794aaccdf169d3737fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } } }
I20260430 20:01:38.641981 12243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:38.642274 12244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f6b46ea13c4794aaccdf169d3737fc. Latest consensus state: current_term: 1 leader_uuid: "94f6b46ea13c4794aaccdf169d3737fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f6b46ea13c4794aaccdf169d3737fc" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35489 } } }
I20260430 20:01:38.642376 12244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:38.643604 12251 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Apr 30 20:01:38 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:38.648849 12251 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:38.653599 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:38.627091 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.10.46.62:35489)
0430 20:01:38.627827 (+ 736us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:38.627850 (+ 23us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:38.627907 (+ 57us) server_negotiation.cc:408] Connection header received
0430 20:01:38.629093 (+ 1186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:38.629125 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:38.629491 (+ 366us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:38.629817 (+ 326us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:38.630941 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.632340 (+ 1399us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:38.634221 (+ 1881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.634643 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:38.645535 (+ 10892us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:38.645564 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:38.645582 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:38.645627 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:38.648864 (+ 3237us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.649851 (+ 987us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.649860 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.649867 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.650012 (+ 145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.650476 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.650480 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.650482 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.650944 (+ 462us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:38.651182 (+ 238us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:38.652332 (+ 1150us) server_negotiation.cc:300] Negotiation successful
0430 20:01:38.652790 (+ 458us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":333,"threads_started":1}
I20260430 20:01:38.657378 12251 catalog_manager.cc:1357] Generated new cluster ID: a83e93e3f53346f78e8c5a2c32d017eb
I20260430 20:01:38.657564 12251 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:38.669649 12251 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:38.670936 12251 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:38.681334 12251 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Generated new TSK 0
I20260430 20:01:38.682343 12251 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:38.753661 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35489
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:38.870405 12265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:38.870720 12265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:38.870827 12265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:38.874423 12265 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:38.874554 12265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:38.874650 12265 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 20:01:38.874694 12265 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:38.874766 12265 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:38.879598 12265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:38.880889 12265 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:38.881902 12265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:38.888744 12270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:38.888772 12273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:38.888944 12271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:38.889189 12265 server_base.cc:1061] running on GCE node
I20260430 20:01:38.889647 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:38.890324 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:38.891608 12265 hybrid_clock.cc:648] HybridClock initialized: now 1777579298891569 us; error 37 us; skew 500 ppm
Apr 30 20:01:38 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:38.894781 12265 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:38.896024 12265 webserver.cc:492] Webserver started at http://127.10.46.1:33299/ using document root <none> and password file <none>
I20260430 20:01:38.896698 12265 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:38.896795 12265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:38.897028 12265 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:38.898803 12265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "7679f60c915e4d1bb9d772235dd3608f"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "a3b04eb3cd4ec2a380cd9c765c47f164"
server_key_iv: "147c563b824ead43a5b8dbd87b688ddb"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.899339 12265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "7679f60c915e4d1bb9d772235dd3608f"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "a3b04eb3cd4ec2a380cd9c765c47f164"
server_key_iv: "147c563b824ead43a5b8dbd87b688ddb"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.903127 12265 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:01:38.906019 12280 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:38.907742 12265 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260430 20:01:38.908031 12265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "7679f60c915e4d1bb9d772235dd3608f"
format_stamp: "Formatted at 2026-04-30 20:01:38 on dist-test-slave-0bdw"
server_key: "a3b04eb3cd4ec2a380cd9c765c47f164"
server_key_iv: "147c563b824ead43a5b8dbd87b688ddb"
server_key_version: "encryptionkey@0"
I20260430 20:01:38.908309 12265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:38.923611 12265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:38.927165 12265 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:38.927471 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:38.929646 12265 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:38.929773 12265 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:38.929855 12265 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:38.929900 12265 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:38.941601 12265 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:34917
Apr 30 20:01:38 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:38.941655 12395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:34917 every 8 connection(s)
I20260430 20:01:38.942874 12265 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:38.946457 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:38.930341 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:58143 (local address 127.10.46.62:35489)
0430 20:01:38.930526 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:38.930531 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:38.931947 (+ 1416us) server_negotiation.cc:408] Connection header received
0430 20:01:38.933605 (+ 1658us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:38.933611 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:38.933687 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:38.933795 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:38.936505 (+ 2710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.937505 (+ 1000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:38.938529 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.938759 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:38.942409 (+ 3650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:38.942442 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:38.942445 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:38.942483 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:38.944442 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.944973 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.944977 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.944978 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.945024 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.945313 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.945317 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.945319 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.945478 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:38.945575 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:38.946123 (+ 548us) server_negotiation.cc:300] Negotiation successful
0430 20:01:38.946287 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:01:38.947108 12289 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:38.930929 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.1:58143)
0430 20:01:38.931647 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:38.931729 (+ 82us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:38.933173 (+ 1444us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:38.934040 (+ 867us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:38.934055 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:38.934883 (+ 828us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:38.935690 (+ 807us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:38.935715 (+ 25us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.937772 (+ 2057us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:38.937791 (+ 19us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:38.938373 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:38.938383 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.938582 (+ 199us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:38.940180 (+ 1598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:38.940210 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:38.942195 (+ 1985us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:38.944585 (+ 2390us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:38.944592 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:38.944604 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:38.944855 (+ 251us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:38.945113 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:38.945117 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:38.945120 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:38.945213 (+ 93us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:38.945590 (+ 377us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:38.945597 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:38.945830 (+ 233us) client_negotiation.cc:770] Sending connection context
0430 20:01:38.946075 (+ 245us) client_negotiation.cc:241] Negotiation successful
0430 20:01:38.946321 (+ 246us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20260430 20:01:38.949344 12396 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35489
I20260430 20:01:38.949755 12396 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:38.950696 12396 heartbeater.cc:507] Master 127.10.46.62:35489 requested a full tablet report, sending...
I20260430 20:01:38.952387 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12265
I20260430 20:01:38.952548 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:38.952821 10424 external_mini_cluster.cc:1468] Setting key 899a6499e764e889aae7b65c766ddb4e
I20260430 20:01:38.953217 12190 ts_manager.cc:194] Registered new tserver with Master: 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917)
I20260430 20:01:38.954854 12190 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:58143
I20260430 20:01:38.962548 12402 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:38.953548 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.1:40305)
0430 20:01:38.953931 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:38.953951 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:38.954137 (+ 186us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:38.954555 (+ 418us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:38.954559 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:38.954780 (+ 221us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:38.955112 (+ 332us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:38.955121 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.956256 (+ 1135us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:38.956261 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:38.956763 (+ 502us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:38.956772 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.956910 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:38.957808 (+ 898us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:38.957823 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:38.958154 (+ 331us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:38.960680 (+ 2526us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:38.960686 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:38.960688 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:38.960983 (+ 295us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:38.961396 (+ 413us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:38.961401 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:38.961403 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:38.961475 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:38.962001 (+ 526us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:38.962006 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:38.962118 (+ 112us) client_negotiation.cc:770] Sending connection context
0430 20:01:38.962230 (+ 112us) client_negotiation.cc:241] Negotiation successful
0430 20:01:38.962387 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":141,"threads_started":1}
I20260430 20:01:38.962672 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:38.953384 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:40305 (local address 127.10.46.62:35489)
0430 20:01:38.953660 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:38.953665 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:38.954092 (+ 427us) server_negotiation.cc:408] Connection header received
0430 20:01:38.954262 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:38.954266 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:38.954329 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:38.954433 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:38.955274 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.956098 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:38.957017 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:38.957858 (+ 841us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:38.958290 (+ 432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:38.958316 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:38.958322 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:38.958353 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:38.960452 (+ 2099us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.961155 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.961161 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.961166 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.961236 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:38.961609 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:38.961615 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:38.961618 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:38.961832 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:38.961954 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:38.962318 (+ 364us) server_negotiation.cc:300] Negotiation successful
0430 20:01:38.962456 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:39.016496 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35489
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:39.140868 12407 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:39.141256 12407 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:39.141559 12407 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:39.146011 12407 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:39.146107 12407 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:39.146178 12407 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 20:01:39.146195 12407 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:39.146238 12407 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:39.151237 12407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:39.152516 12407 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:39.153530 12407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:39.161957 12413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:39.162014 12412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:39.161957 12415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:39.162372 12407 server_base.cc:1061] running on GCE node
I20260430 20:01:39.163097 12407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:39.164160 12407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:39.165422 12407 hybrid_clock.cc:648] HybridClock initialized: now 1777579299165361 us; error 96 us; skew 500 ppm
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:39.170913 12407 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:39.173009 12407 webserver.cc:492] Webserver started at http://127.10.46.2:35685/ using document root <none> and password file <none>
I20260430 20:01:39.173774 12407 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:39.173899 12407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:39.174417 12407 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:39.176959 12407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "f636f078b88a4cffb379cc128892904a"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "9e3eeeaa82dabdb4a7a9ea6f3032f972"
server_key_iv: "c18049b2d1032a91b8ed690662663b19"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.177548 12407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "f636f078b88a4cffb379cc128892904a"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "9e3eeeaa82dabdb4a7a9ea6f3032f972"
server_key_iv: "c18049b2d1032a91b8ed690662663b19"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.181329 12407 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:01:39.184018 12422 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:39.185438 12407 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:39.185560 12407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "f636f078b88a4cffb379cc128892904a"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "9e3eeeaa82dabdb4a7a9ea6f3032f972"
server_key_iv: "c18049b2d1032a91b8ed690662663b19"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.185706 12407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:39.201748 12407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:39.205485 12407 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:39.205717 12407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:39.207346 12407 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:39.207402 12407 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:39.207448 12407 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:39.207504 12407 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:39.222113 12407 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:35061
I20260430 20:01:39.222134 12537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:35061 every 8 connection(s)
I20260430 20:01:39.223126 12407 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:39.223210 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.208333 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:38621 (local address 127.10.46.62:35489)
0430 20:01:39.208480 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.208484 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.209442 (+ 958us) server_negotiation.cc:408] Connection header received
0430 20:01:39.210435 (+ 993us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.210439 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.210503 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.210601 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.212270 (+ 1669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.212944 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.213949 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.214422 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.218579 (+ 4157us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.218606 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.218609 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.218649 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.220922 (+ 2273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.221618 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.221622 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.221624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.221682 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.222072 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.222074 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.222076 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.222283 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.222395 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.222893 (+ 498us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.223044 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 20:01:39.223973 12431 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.208639 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.2:38621)
0430 20:01:39.209151 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.209186 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.210227 (+ 1041us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.210768 (+ 541us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.210780 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.211362 (+ 582us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:39.212103 (+ 741us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.212117 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.213104 (+ 987us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.213110 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.213804 (+ 694us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.213817 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.214068 (+ 251us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.215178 (+ 1110us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:39.215222 (+ 44us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:39.218261 (+ 3039us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:39.221066 (+ 2805us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.221073 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.221088 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.221477 (+ 389us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.221804 (+ 327us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.221809 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.221811 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.221929 (+ 118us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.222407 (+ 478us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:39.222415 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:39.222649 (+ 234us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.222851 (+ 202us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.223093 (+ 242us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":155,"threads_started":1}
I20260430 20:01:39.225087 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12407
I20260430 20:01:39.225224 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:39.225229 12538 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35489
I20260430 20:01:39.225464 10424 external_mini_cluster.cc:1468] Setting key b414c480a8f0979e8d83c0451a18d358
I20260430 20:01:39.225479 12538 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:39.226087 12538 heartbeater.cc:507] Master 127.10.46.62:35489 requested a full tablet report, sending...
I20260430 20:01:39.227246 12191 ts_manager.cc:194] Registered new tserver with Master: f636f078b88a4cffb379cc128892904a (127.10.46.2:35061)
I20260430 20:01:39.228432 12191 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:38621
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:39.236652 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.228000 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:44473 (local address 127.10.46.62:35489)
0430 20:01:39.228240 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.228245 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.228958 (+ 713us) server_negotiation.cc:408] Connection header received
0430 20:01:39.229337 (+ 379us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.229341 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.229399 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.229444 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.230166 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.230866 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.231847 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.232007 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.233078 (+ 1071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.233096 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.233098 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.233122 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.234545 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.235092 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.235095 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.235097 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.235144 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.235433 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.235437 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.235438 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.235619 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.235710 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.236416 (+ 706us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.236533 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 20:01:39.236661 12544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.228380 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.2:44473)
0430 20:01:39.228788 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.228806 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.229136 (+ 330us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.229562 (+ 426us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.229565 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.229765 (+ 200us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:39.230039 (+ 274us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.230050 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.230994 (+ 944us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.231001 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.231645 (+ 644us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.231656 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.231829 (+ 173us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.232419 (+ 590us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:39.232442 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:39.232916 (+ 474us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:39.234674 (+ 1758us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.234680 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.234683 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.234948 (+ 265us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.235252 (+ 304us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.235255 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.235258 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.235318 (+ 60us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.235866 (+ 548us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:39.235872 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:39.236047 (+ 175us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.236344 (+ 297us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.236506 (+ 162us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:39.282811 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35489
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:39.393055 12549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:39.393476 12549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:39.393648 12549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:39.397971 12549 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:39.398067 12549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:39.398125 12549 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 20:01:39.398142 12549 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:01:39.398186 12549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:39.404250 12549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:39.406038 12549 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:39.407433 12549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:39.416460 12555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:39.416440 12557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:39.416479 12554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:39.417147 12549 server_base.cc:1061] running on GCE node
I20260430 20:01:39.417840 12549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:39.418678 12549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:39.419903 12549 hybrid_clock.cc:648] HybridClock initialized: now 1777579299419905 us; error 68 us; skew 500 ppm
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:39.423035 12549 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:39.424340 12549 webserver.cc:492] Webserver started at http://127.10.46.3:45909/ using document root <none> and password file <none>
I20260430 20:01:39.424896 12549 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:39.424943 12549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:39.425123 12549 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:39.427085 12549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "f1fb83251ca0481a8668660e3299c3a6"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "35070c39f432d4bc00a859dffb644935"
server_key_iv: "71b1781ae45bbb301bc757a45ffc9b6c"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.427572 12549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "f1fb83251ca0481a8668660e3299c3a6"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "35070c39f432d4bc00a859dffb644935"
server_key_iv: "71b1781ae45bbb301bc757a45ffc9b6c"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.432983 12549 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260430 20:01:39.436069 12564 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:39.437562 12549 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260430 20:01:39.437690 12549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "f1fb83251ca0481a8668660e3299c3a6"
format_stamp: "Formatted at 2026-04-30 20:01:39 on dist-test-slave-0bdw"
server_key: "35070c39f432d4bc00a859dffb644935"
server_key_iv: "71b1781ae45bbb301bc757a45ffc9b6c"
server_key_version: "encryptionkey@0"
I20260430 20:01:39.437793 12549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:39.453907 12549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:39.457176 12549 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:39.457363 12549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:39.458930 12549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:39.459022 12549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:39.459080 12549 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:39.459131 12549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:39.471829 12549 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:43287
I20260430 20:01:39.472030 12679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:43287 every 8 connection(s)
I20260430 20:01:39.472910 12549 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:39.474890 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.459709 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:51705 (local address 127.10.46.62:35489)
0430 20:01:39.459887 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.459892 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.460734 (+ 842us) server_negotiation.cc:408] Connection header received
0430 20:01:39.462178 (+ 1444us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.462182 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.462292 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.462395 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.464728 (+ 2333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.465757 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.466866 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.467080 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.470504 (+ 3424us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.470536 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.470540 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.470580 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.472712 (+ 2132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.473324 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.473327 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.473328 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.473400 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.473727 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.473746 (+ 19us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.473751 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.473952 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.474047 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.474656 (+ 609us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.474763 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260430 20:01:39.475433 12573 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.460058 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.3:51705)
0430 20:01:39.460559 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.460605 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.461773 (+ 1168us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.462686 (+ 913us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.462697 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.463439 (+ 742us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:39.464441 (+ 1002us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.464470 (+ 29us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.466059 (+ 1589us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.466065 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.466710 (+ 645us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.466725 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.466997 (+ 272us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.467744 (+ 747us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:39.467810 (+ 66us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:39.470304 (+ 2494us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:39.472855 (+ 2551us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.472861 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.472872 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.473198 (+ 326us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.473507 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.473510 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.473512 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.473619 (+ 107us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.474114 (+ 495us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:39.474121 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:39.474386 (+ 265us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.474554 (+ 168us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.474752 (+ 198us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260430 20:01:39.476792 12680 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35489
I20260430 20:01:39.476986 12680 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:39.477517 12680 heartbeater.cc:507] Master 127.10.46.62:35489 requested a full tablet report, sending...
I20260430 20:01:39.478725 12190 ts_manager.cc:194] Registered new tserver with Master: f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:39.479508 12190 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:51705
I20260430 20:01:39.479813 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12549
I20260430 20:01:39.479938 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:39.480233 10424 external_mini_cluster.cc:1468] Setting key 1f2d2613de18fe962a8273f5d14e631f
I20260430 20:01:39.483456 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:01:39.487246 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.479366 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:58621 (local address 127.10.46.62:35489)
0430 20:01:39.479506 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.479510 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.479527 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:01:39.479570 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.479573 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.479617 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.479700 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.480439 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.481152 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.481966 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.482147 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.483360 (+ 1213us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.483378 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.483381 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.483412 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.485212 (+ 1800us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.485958 (+ 746us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.485965 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.485969 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.486054 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.486334 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.486338 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.486341 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.486565 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.486675 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.486996 (+ 321us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.487119 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 20:01:39.487495 12686 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.478980 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.10.46.3:58621)
0430 20:01:39.479326 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.479343 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.479448 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.479722 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.479726 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.479998 (+ 272us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:39.480305 (+ 307us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.480313 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.481302 (+ 989us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.481305 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.481843 (+ 538us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.481856 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.481986 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.482612 (+ 626us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:39.482632 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:39.483181 (+ 549us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:39.485382 (+ 2201us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.485390 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.485396 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.485688 (+ 292us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.486176 (+ 488us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.486179 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.486180 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.486232 (+ 52us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.486683 (+ 451us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:39.486686 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:39.486808 (+ 122us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.486937 (+ 129us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.487105 (+ 168us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:39.512689 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.503503 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.10.46.62:35489)
0430 20:01:39.503648 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.503651 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.503665 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:01:39.503717 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.503720 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.503824 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.503928 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.505001 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.505534 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.506488 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.506635 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.509583 (+ 2948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.509609 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.509611 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.509645 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.511139 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.511566 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.511568 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.511569 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.511612 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.511901 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.511906 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.511908 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.512076 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.512210 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.512446 (+ 236us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.512549 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260430 20:01:39.515666 12190 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:01:39.518362 12190 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:39.534152 12697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.528349 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.0.0.1:55020)
0430 20:01:39.528735 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.528777 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.528916 (+ 139us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.529618 (+ 702us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.529623 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.529652 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.530213 (+ 561us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.530232 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.532066 (+ 1834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.532071 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.533183 (+ 1112us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.533194 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.533408 (+ 214us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.533488 (+ 80us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.533584 (+ 96us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.533928 (+ 344us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
I20260430 20:01:39.534564 12698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.528523 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.10.46.1:34917)
0430 20:01:39.529099 (+ 576us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.529106 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.529129 (+ 23us) server_negotiation.cc:408] Connection header received
0430 20:01:39.529231 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.529236 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.529421 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.529593 (+ 172us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.530460 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.531918 (+ 1458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.533431 (+ 1513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.534117 (+ 686us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.534246 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.534330 (+ 84us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.534421 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":92,"threads_started":1}
I20260430 20:01:39.534443 12701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.529093 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:43287 (local address 127.0.0.1:47918)
0430 20:01:39.529864 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.529878 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.529983 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.530937 (+ 954us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.530942 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.530977 (+ 35us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.531429 (+ 452us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.531444 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.533054 (+ 1610us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.533059 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.533973 (+ 914us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.533984 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.534111 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.534126 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.534175 (+ 49us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.534250 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":107,"threads_started":1}
I20260430 20:01:39.535647 12700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.529300 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.10.46.3:43287)
0430 20:01:39.530247 (+ 947us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.530255 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.530277 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:01:39.530361 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.530365 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.530536 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.530976 (+ 440us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.531594 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.532923 (+ 1329us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.534404 (+ 1481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.535188 (+ 784us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.535317 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.535391 (+ 74us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.535491 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":121,"threads_started":1}
I20260430 20:01:39.536054 12699 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.529405 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.0.0.1:43332)
0430 20:01:39.530214 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.530227 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.530311 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.531259 (+ 948us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.531263 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.531288 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.531670 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.531681 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.533474 (+ 1793us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.533477 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.535596 (+ 2119us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.535606 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.535736 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.535757 (+ 21us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.535852 (+ 95us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.535921 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":99,"threads_started":1}
I20260430 20:01:39.537123 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.529849 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.10.46.2:35061)
0430 20:01:39.530393 (+ 544us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.530403 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.530435 (+ 32us) server_negotiation.cc:408] Connection header received
0430 20:01:39.530549 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.530553 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.530930 (+ 377us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.531141 (+ 211us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.531871 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.533313 (+ 1442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.536269 (+ 2956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.536828 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.536903 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.536960 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.537017 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":132,"threads_started":1}
I20260430 20:01:39.538137 12614 tablet_service.cc:1511] Processing CreateTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da (DEFAULT_TABLE table=test-table [id=361e94792fc548438569adaf1da58723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.538122 12330 tablet_service.cc:1511] Processing CreateTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da (DEFAULT_TABLE table=test-table [id=361e94792fc548438569adaf1da58723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.539170 12614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7818fb0e0cd647f2a8ce023822eff1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.539170 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7818fb0e0cd647f2a8ce023822eff1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.539695 12472 tablet_service.cc:1511] Processing CreateTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da (DEFAULT_TABLE table=test-table [id=361e94792fc548438569adaf1da58723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.540588 12472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7818fb0e0cd647f2a8ce023822eff1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.545674 12703 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Bootstrap starting.
I20260430 20:01:39.546532 12705 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Bootstrap starting.
I20260430 20:01:39.546890 12704 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Bootstrap starting.
I20260430 20:01:39.548235 12703 tablet_bootstrap.cc:654] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.548578 12704 tablet_bootstrap.cc:654] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.548607 12705 tablet_bootstrap.cc:654] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.549198 12703 log.cc:826] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:39.549628 12704 log.cc:826] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:39.549798 12705 log.cc:826] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:39.551326 12704 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: No bootstrap required, opened a new log
I20260430 20:01:39.551340 12703 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: No bootstrap required, opened a new log
I20260430 20:01:39.551544 12704 ts_tablet_manager.cc:1403] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:01:39.551543 12703 ts_tablet_manager.cc:1403] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260430 20:01:39.551563 12705 tablet_bootstrap.cc:492] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: No bootstrap required, opened a new log
I20260430 20:01:39.551857 12705 ts_tablet_manager.cc:1403] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:01:39.554260 12705 raft_consensus.cc:359] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.554491 12705 raft_consensus.cc:385] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.554603 12705 raft_consensus.cc:740] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636f078b88a4cffb379cc128892904a, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.554682 12704 raft_consensus.cc:359] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.554680 12703 raft_consensus.cc:359] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.554883 12703 raft_consensus.cc:385] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.554883 12704 raft_consensus.cc:385] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.554934 12704 raft_consensus.cc:740] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.554935 12703 raft_consensus.cc:740] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.555240 12705 consensus_queue.cc:260] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.555337 12704 consensus_queue.cc:260] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.555337 12703 consensus_queue.cc:260] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.556031 12538 heartbeater.cc:499] Master 127.10.46.62:35489 was elected leader, sending a full tablet report...
I20260430 20:01:39.556308 12703 ts_tablet_manager.cc:1434] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260430 20:01:39.556648 12396 heartbeater.cc:499] Master 127.10.46.62:35489 was elected leader, sending a full tablet report...
I20260430 20:01:39.556792 12704 ts_tablet_manager.cc:1434] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260430 20:01:39.557106 12680 heartbeater.cc:499] Master 127.10.46.62:35489 was elected leader, sending a full tablet report...
I20260430 20:01:39.558770 12705 ts_tablet_manager.cc:1434] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260430 20:01:39.598600 12711 raft_consensus.cc:493] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:39.598843 12711 raft_consensus.cc:515] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.600416 12711 leader_election.cc:290] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f636f078b88a4cffb379cc128892904a (127.10.46.2:35061)
I20260430 20:01:39.605237 12573 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.600651 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.10.46.3:54681)
0430 20:01:39.600794 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.600813 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.600940 (+ 127us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.601812 (+ 872us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.601815 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.601835 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.602136 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.602147 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.603847 (+ 1700us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.603851 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.604797 (+ 946us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.604808 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.604942 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.604961 (+ 19us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.605025 (+ 64us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.605087 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260430 20:01:39.605696 12698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.600993 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:54681 (local address 127.10.46.1:34917)
0430 20:01:39.601539 (+ 546us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.601544 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.601559 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:01:39.601626 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.601632 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.601690 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.601784 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.602290 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.603691 (+ 1401us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.604920 (+ 1229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.605425 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.605474 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.605529 (+ 55us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.605581 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260430 20:01:39.606042 12712 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.600996 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.10.46.3:59035)
0430 20:01:39.601633 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.601649 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.601749 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.602457 (+ 708us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.602463 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.602489 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.602850 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.602858 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.604006 (+ 1148us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.604010 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.604838 (+ 828us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.604852 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.605701 (+ 849us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.605714 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.605808 (+ 94us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.605869 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":168,"threads_started":1}
I20260430 20:01:39.606046 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.601934 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:59035 (local address 127.10.46.2:35061)
0430 20:01:39.602095 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.602099 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.602114 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:01:39.602170 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.602175 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.602262 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.602344 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.603014 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.603857 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.605015 (+ 1158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.605610 (+ 595us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.605643 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.605857 (+ 214us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.605932 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 20:01:39.606366 12350 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7818fb0e0cd647f2a8ce023822eff1da" candidate_uuid: "f1fb83251ca0481a8668660e3299c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f" is_pre_election: true
I20260430 20:01:39.606724 12350 raft_consensus.cc:2468] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fb83251ca0481a8668660e3299c3a6 in term 0.
I20260430 20:01:39.607041 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7818fb0e0cd647f2a8ce023822eff1da" candidate_uuid: "f1fb83251ca0481a8668660e3299c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a" is_pre_election: true
I20260430 20:01:39.607249 12568 leader_election.cc:304] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f1fb83251ca0481a8668660e3299c3a6; no voters:
I20260430 20:01:39.607287 12492 raft_consensus.cc:2468] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fb83251ca0481a8668660e3299c3a6 in term 0.
I20260430 20:01:39.607590 12711 raft_consensus.cc:2804] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:39.607672 12711 raft_consensus.cc:493] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:39.607726 12711 raft_consensus.cc:3060] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.608907 12711 raft_consensus.cc:515] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.609324 12711 leader_election.cc:290] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [CANDIDATE]: Term 1 election: Requested vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f636f078b88a4cffb379cc128892904a (127.10.46.2:35061)
I20260430 20:01:39.609862 12350 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7818fb0e0cd647f2a8ce023822eff1da" candidate_uuid: "f1fb83251ca0481a8668660e3299c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f"
I20260430 20:01:39.610000 12350 raft_consensus.cc:3060] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.610958 12350 raft_consensus.cc:2468] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fb83251ca0481a8668660e3299c3a6 in term 1.
I20260430 20:01:39.611244 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7818fb0e0cd647f2a8ce023822eff1da" candidate_uuid: "f1fb83251ca0481a8668660e3299c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a"
I20260430 20:01:39.611286 12568 leader_election.cc:304] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f1fb83251ca0481a8668660e3299c3a6; no voters:
I20260430 20:01:39.611403 12492 raft_consensus.cc:3060] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.611495 12711 raft_consensus.cc:2804] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:39.611756 12711 raft_consensus.cc:697] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 1 LEADER]: Becoming Leader. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Running, Role: LEADER
I20260430 20:01:39.612154 12711 consensus_queue.cc:237] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } }
I20260430 20:01:39.612763 12492 raft_consensus.cc:2468] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fb83251ca0481a8668660e3299c3a6 in term 1.
I20260430 20:01:39.616859 12191 catalog_manager.cc:5671] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3). New cstate: current_term: 1 leader_uuid: "f1fb83251ca0481a8668660e3299c3a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } health_report { overall_health: UNKNOWN } } }
Apr 30 20:01:39 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:39.658519 12720 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.645791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35489 (local address 127.0.0.1:34804)
0430 20:01:39.646162 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.646177 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.646318 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.646847 (+ 529us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.646850 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.647348 (+ 498us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:39.647650 (+ 302us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.647660 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.648763 (+ 1103us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.648769 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.649313 (+ 544us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.649324 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.649519 (+ 195us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.650323 (+ 804us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:39.650338 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:39.653244 (+ 2906us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:39.656076 (+ 2832us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.656090 (+ 14us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.656094 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.656421 (+ 327us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.657056 (+ 635us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:39.657059 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:39.657061 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:39.657157 (+ 96us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:39.657717 (+ 560us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:39.657724 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:39.657947 (+ 223us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.658093 (+ 146us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.658322 (+ 229us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260430 20:01:39.658877 12240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.645913 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.10.46.62:35489)
0430 20:01:39.646058 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.646063 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.646288 (+ 225us) server_negotiation.cc:408] Connection header received
0430 20:01:39.646401 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.646404 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.646451 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.646528 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:39.647928 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.648426 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.649527 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.649688 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.653640 (+ 3952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:39.653666 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:39.653668 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:39.653702 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:39.655732 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.656597 (+ 865us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.656601 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.656603 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.656732 (+ 129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:39.657360 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:39.657363 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:39.657365 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:39.657541 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:39.657638 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.658194 (+ 556us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.658470 (+ 276us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":57}
I20260430 20:01:39.665366 12191 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:34804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 20:01:39.666198 12191 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:39.673475 12330 tablet_service.cc:1511] Processing CreateTablet for tablet b88bee9eabde40f2a80171baccf9eb7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa8b7868ec584822910d239d21378595]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:39.674595 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88bee9eabde40f2a80171baccf9eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.673992 12472 tablet_service.cc:1511] Processing CreateTablet for tablet b88bee9eabde40f2a80171baccf9eb7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa8b7868ec584822910d239d21378595]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:39.675086 12472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88bee9eabde40f2a80171baccf9eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.674794 12614 tablet_service.cc:1511] Processing CreateTablet for tablet b88bee9eabde40f2a80171baccf9eb7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa8b7868ec584822910d239d21378595]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:01:39.675510 12614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88bee9eabde40f2a80171baccf9eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.677317 12703 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: Bootstrap starting.
I20260430 20:01:39.678140 12705 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: Bootstrap starting.
I20260430 20:01:39.678438 12704 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: Bootstrap starting.
I20260430 20:01:39.678604 12703 tablet_bootstrap.cc:654] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.679217 12705 tablet_bootstrap.cc:654] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.679543 12704 tablet_bootstrap.cc:654] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.680027 12703 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: No bootstrap required, opened a new log
I20260430 20:01:39.680142 12703 ts_tablet_manager.cc:1403] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:39.680619 12705 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: No bootstrap required, opened a new log
I20260430 20:01:39.680711 12705 ts_tablet_manager.cc:1403] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260430 20:01:39.680787 12703 raft_consensus.cc:359] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.680914 12703 raft_consensus.cc:385] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.680949 12703 raft_consensus.cc:740] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.681111 12703 consensus_queue.cc:260] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.681332 12704 tablet_bootstrap.cc:492] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: No bootstrap required, opened a new log
I20260430 20:01:39.681406 12703 ts_tablet_manager.cc:1434] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:39.681419 12704 ts_tablet_manager.cc:1403] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:39.681751 12710 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.681281 12705 raft_consensus.cc:359] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.681874 12710 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.681860 12705 raft_consensus.cc:385] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.681906 12705 raft_consensus.cc:740] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636f078b88a4cffb379cc128892904a, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.681967 12704 raft_consensus.cc:359] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.682045 12705 consensus_queue.cc:260] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.682125 12704 raft_consensus.cc:385] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.682196 12704 raft_consensus.cc:740] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.682353 12705 ts_tablet_manager.cc:1434] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260430 20:01:39.682333 12704 consensus_queue.cc:260] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.682587 12704 ts_tablet_manager.cc:1434] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:39.682647 12709 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.682791 12709 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.683563 12711 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.683689 12711 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260430 20:01:39.694425 12397 tablet.cc:2404] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:01:39.723973 12681 tablet.cc:2404] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:01:39.724452 12539 tablet.cc:2404] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:01:39.809367 12709 raft_consensus.cc:493] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:39.809546 12709 raft_consensus.cc:515] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.810842 12709 leader_election.cc:290] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:39.814556 12724 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.811116 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.10.46.2:55785)
0430 20:01:39.811466 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.811479 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.811626 (+ 147us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.811949 (+ 323us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.811952 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.812016 (+ 64us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.812332 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.812344 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.813289 (+ 945us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.813292 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.814119 (+ 827us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.814129 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.814250 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.814270 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.814323 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.814385 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":2304,"thread_start_us":117,"threads_started":1}
I20260430 20:01:39.814975 12698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.811229 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:55785 (local address 127.10.46.1:34917)
0430 20:01:39.811383 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.811388 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.811564 (+ 176us) server_negotiation.cc:408] Connection header received
0430 20:01:39.811739 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.811742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.811826 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.811913 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.812468 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.813160 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.814226 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.814690 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.814733 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.814797 (+ 64us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.814849 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 20:01:39.815040 12725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.811521 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:43287 (local address 127.10.46.2:37407)
0430 20:01:39.811910 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.811923 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.812027 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.812311 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.812314 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.812331 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:39.812590 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.812597 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.813512 (+ 915us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.813515 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.814292 (+ 777us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.814300 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.814508 (+ 208us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.814524 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.814579 (+ 55us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.814630 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"spinlock_wait_cycles":3456,"thread_start_us":76,"threads_started":1}
I20260430 20:01:39.815301 12700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.811680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:37407 (local address 127.10.46.3:43287)
0430 20:01:39.811862 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.811866 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.811996 (+ 130us) server_negotiation.cc:408] Connection header received
0430 20:01:39.812132 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.812136 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.812190 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.812289 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:39.812813 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.813393 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.814491 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.814986 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.815019 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.815077 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.815138 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260430 20:01:39.815392 12350 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88bee9eabde40f2a80171baccf9eb7b" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f" is_pre_election: true
I20260430 20:01:39.815567 12350 raft_consensus.cc:2468] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 0.
I20260430 20:01:39.815877 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88bee9eabde40f2a80171baccf9eb7b" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6" is_pre_election: true
I20260430 20:01:39.815953 12426 leader_election.cc:304] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:39.816087 12634 raft_consensus.cc:2468] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 0.
I20260430 20:01:39.816280 12709 raft_consensus.cc:2804] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:39.816345 12709 raft_consensus.cc:493] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:39.816396 12709 raft_consensus.cc:3060] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.817323 12709 raft_consensus.cc:515] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.817674 12709 leader_election.cc:290] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 election: Requested vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:39.818106 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88bee9eabde40f2a80171baccf9eb7b" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6"
I20260430 20:01:39.818106 12350 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88bee9eabde40f2a80171baccf9eb7b" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f"
I20260430 20:01:39.818241 12350 raft_consensus.cc:3060] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.818238 12634 raft_consensus.cc:3060] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:39.819329 12350 raft_consensus.cc:2468] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 1.
I20260430 20:01:39.819326 12634 raft_consensus.cc:2468] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 1.
I20260430 20:01:39.819826 12426 leader_election.cc:304] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1fb83251ca0481a8668660e3299c3a6, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:39.820017 12709 raft_consensus.cc:2804] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:39.820264 12709 raft_consensus.cc:697] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [term 1 LEADER]: Becoming Leader. State: Replica: f636f078b88a4cffb379cc128892904a, State: Running, Role: LEADER
I20260430 20:01:39.820689 12709 consensus_queue.cc:237] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.823283 12726 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: TxnStatusTablet state changed. Reason: New leader f636f078b88a4cffb379cc128892904a. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.823432 12726 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:39.823755 12728 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 20:01:39.824003 12728 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 20:01:39.824673 12190 catalog_manager.cc:5671] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a reported cstate change: term changed from 0 to 1, leader changed from <none> to f636f078b88a4cffb379cc128892904a (127.10.46.2). New cstate: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:39.840920 12720 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.837313 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.0.0.1:43342)
0430 20:01:39.837520 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.837535 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.837639 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.838011 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.838014 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.838023 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:39.838218 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.838224 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.839226 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.839230 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.839655 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.839661 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.839824 (+ 163us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.839876 (+ 52us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.840637 (+ 761us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.840648 (+ 11us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.840718 (+ 70us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.840791 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260430 20:01:39.840929 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.837437 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.10.46.2:35061)
0430 20:01:39.837564 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.837569 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.837586 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:01:39.837776 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.837780 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.837846 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.837944 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:39.838334 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.839087 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.839790 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.839968 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.840028 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.840512 (+ 484us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.840624 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.840747 (+ 123us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.840798 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260430 20:01:39.846261 12634 raft_consensus.cc:1275] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Refusing update from remote peer f636f078b88a4cffb379cc128892904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:39.846392 12350 raft_consensus.cc:1275] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Refusing update from remote peer f636f078b88a4cffb379cc128892904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:39.846954 12726 consensus_queue.cc:1048] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:39.847169 12709 consensus_queue.cc:1048] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:39.851126 12711 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: TxnStatusTablet state changed. Reason: New leader f636f078b88a4cffb379cc128892904a. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.851291 12711 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.852421 12710 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: TxnStatusTablet state changed. Reason: New leader f636f078b88a4cffb379cc128892904a. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.852560 12710 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.856067 12731 mvcc.cc:204] Tried to move back new op lower bound from 7280964812163641344 to 7280964812068638720. Current Snapshot: MvccSnapshot[applied={T|T < 7280964812163641344}]
I20260430 20:01:39.856244 12709 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.856374 12709 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:39.856504 12709 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.856570 12709 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f636f078b88a4cffb379cc128892904a: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:01:39.857319 12710 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.857432 12710 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P 7679f60c915e4d1bb9d772235dd3608f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.858608 12711 tablet_replica.cc:442] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } } }
I20260430 20:01:39.858726 12711 tablet_replica.cc:445] T b88bee9eabde40f2a80171baccf9eb7b P f1fb83251ca0481a8668660e3299c3a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:01:39.860121 12733 mvcc.cc:204] Tried to move back new op lower bound from 7280964812163641344 to 7280964812068638720. Current Snapshot: MvccSnapshot[applied={T|T < 7280964812163641344}]
I20260430 20:01:39.874083 12700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.870523 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.10.46.3:43287)
0430 20:01:39.870677 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.870682 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.870698 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:01:39.870789 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.870793 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.870861 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.870968 (+ 107us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:39.871517 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.872296 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.872935 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.873126 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.873184 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.873682 (+ 498us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.873789 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.873887 (+ 98us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.873938 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 20:01:39.886296 12686 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.882740 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.10.46.3:40793)
0430 20:01:39.882919 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.882933 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.883022 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.883583 (+ 561us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.883586 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.883594 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:39.883821 (+ 227us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.883826 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.884871 (+ 1045us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.884878 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.885448 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.885456 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.885562 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.885599 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.886031 (+ 432us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.886036 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.886106 (+ 70us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.886160 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260430 20:01:39.886319 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.883182 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:40793 (local address 127.10.46.2:35061)
0430 20:01:39.883333 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.883337 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.883353 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:01:39.883406 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.883410 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.883470 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.883558 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:39.883935 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.884709 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.885564 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.885767 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.885810 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.885910 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.885992 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.886133 (+ 141us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.886185 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:01:39.893985 12350 raft_consensus.cc:1275] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Refusing update from remote peer f1fb83251ca0481a8668660e3299c3a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:39.894120 12492 raft_consensus.cc:1275] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Refusing update from remote peer f1fb83251ca0481a8668660e3299c3a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:39.894546 12711 consensus_queue.cc:1048] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:39.894775 12713 consensus_queue.cc:1048] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:39.922961 12746 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.918330 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:43287 (local address 127.10.46.2:47651)
0430 20:01:39.918875 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:39.918892 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:39.919018 (+ 126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:39.919480 (+ 462us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:39.919483 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:39.919494 (+ 11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:39.920288 (+ 794us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.920297 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.921317 (+ 1020us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.921321 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:39.921923 (+ 602us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:39.921932 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.922073 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.922118 (+ 45us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.922643 (+ 525us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.922651 (+ 8us) client_negotiation.cc:770] Sending connection context
0430 20:01:39.922725 (+ 74us) client_negotiation.cc:241] Negotiation successful
0430 20:01:39.922789 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":116,"threads_started":1}
I20260430 20:01:39.922931 12700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:39.918517 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:47651 (local address 127.10.46.3:43287)
0430 20:01:39.918661 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:39.918668 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:39.918985 (+ 317us) server_negotiation.cc:408] Connection header received
0430 20:01:39.919131 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:39.919134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:39.919195 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:39.919280 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:39.920410 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.921168 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:39.922074 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:39.922266 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:39.922312 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:39.922397 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:39.922493 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:39.922741 (+ 248us) server_negotiation.cc:300] Negotiation successful
0430 20:01:39.922786 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 20:01:39.951584 12191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:39.951805 12191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:39.954638 12191 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Sending DeleteTablet for 3 replicas of tablet 7818fb0e0cd647f2a8ce023822eff1da
I20260430 20:01:39.955760 12472 tablet_service.cc:1558] Processing DeleteTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:39 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260430 20:01:39.955907 12330 tablet_service.cc:1558] Processing DeleteTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:39 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:55020
I20260430 20:01:39.956638 12191 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 20:01:39.956693 12614 tablet_service.cc:1558] Processing DeleteTablet for tablet 7818fb0e0cd647f2a8ce023822eff1da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:39 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:47918
W20260430 20:01:39.957099 12191 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:39.957427 12752 tablet_replica.cc:333] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: stopping tablet replica
I20260430 20:01:39.957773 12752 raft_consensus.cc:2243] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:01:39.958268 12752 raft_consensus.cc:2272] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:39.963483 12330 tablet_service.cc:1511] Processing CreateTablet for tablet 6019a52c35f54d03b74e8af9941cb08e (DEFAULT_TABLE table=test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.963816 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6019a52c35f54d03b74e8af9941cb08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.963814 12472 tablet_service.cc:1511] Processing CreateTablet for tablet 6019a52c35f54d03b74e8af9941cb08e (DEFAULT_TABLE table=test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.964104 12472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6019a52c35f54d03b74e8af9941cb08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.966228 12703 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Bootstrap starting.
I20260430 20:01:39.966564 12614 tablet_service.cc:1511] Processing CreateTablet for tablet 6019a52c35f54d03b74e8af9941cb08e (DEFAULT_TABLE table=test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:39.966833 12614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6019a52c35f54d03b74e8af9941cb08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:39.967196 12703 tablet_bootstrap.cc:654] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.969118 12704 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Bootstrap starting.
I20260430 20:01:39.969316 12705 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Bootstrap starting.
I20260430 20:01:39.970212 12704 tablet_bootstrap.cc:654] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.970270 12705 tablet_bootstrap.cc:654] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:39.970929 12750 tablet_replica.cc:333] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: stopping tablet replica
I20260430 20:01:39.971223 12750 raft_consensus.cc:2243] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:39.971418 12750 raft_consensus.cc:2272] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:39.972481 12704 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: No bootstrap required, opened a new log
I20260430 20:01:39.972580 12704 ts_tablet_manager.cc:1403] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 20:01:39.972855 12705 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: No bootstrap required, opened a new log
I20260430 20:01:39.972945 12705 ts_tablet_manager.cc:1403] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260430 20:01:39.973134 12704 raft_consensus.cc:359] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.973254 12704 raft_consensus.cc:385] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.973290 12704 raft_consensus.cc:740] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.973340 12705 raft_consensus.cc:359] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.973433 12705 raft_consensus.cc:385] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.973465 12705 raft_consensus.cc:740] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636f078b88a4cffb379cc128892904a, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.973426 12704 consensus_queue.cc:260] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.973575 12705 consensus_queue.cc:260] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.973701 12704 ts_tablet_manager.cc:1434] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:01:39.973811 12705 ts_tablet_manager.cc:1434] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:39.974086 12750 ts_tablet_manager.cc:1916] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:39.979074 12752 ts_tablet_manager.cc:1916] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:39.981355 12750 ts_tablet_manager.cc:1929] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:39.981765 12751 tablet_replica.cc:333] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: stopping tablet replica
I20260430 20:01:39.982102 12751 raft_consensus.cc:2243] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:39.982405 12751 raft_consensus.cc:2272] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:39.982545 12750 log.cc:1199] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/7818fb0e0cd647f2a8ce023822eff1da
I20260430 20:01:39.983090 12750 ts_tablet_manager.cc:1950] T 7818fb0e0cd647f2a8ce023822eff1da P f636f078b88a4cffb379cc128892904a: Deleting consensus metadata
I20260430 20:01:39.983093 12752 ts_tablet_manager.cc:1929] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:39.983180 12752 log.cc:1199] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/7818fb0e0cd647f2a8ce023822eff1da
I20260430 20:01:39.983500 12752 ts_tablet_manager.cc:1950] T 7818fb0e0cd647f2a8ce023822eff1da P f1fb83251ca0481a8668660e3299c3a6: Deleting consensus metadata
I20260430 20:01:39.984599 12178 catalog_manager.cc:5002] TS f636f078b88a4cffb379cc128892904a (127.10.46.2:35061): tablet 7818fb0e0cd647f2a8ce023822eff1da (table test-table [id=361e94792fc548438569adaf1da58723]) successfully deleted
I20260430 20:01:39.985589 12178 catalog_manager.cc:5002] TS f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287): tablet 7818fb0e0cd647f2a8ce023822eff1da (table test-table [id=361e94792fc548438569adaf1da58723]) successfully deleted
I20260430 20:01:39.986786 12703 tablet_bootstrap.cc:492] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: No bootstrap required, opened a new log
I20260430 20:01:39.986902 12703 ts_tablet_manager.cc:1403] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20260430 20:01:39.987478 12703 raft_consensus.cc:359] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.987615 12703 raft_consensus.cc:385] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:39.987653 12703 raft_consensus.cc:740] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Initialized, Role: FOLLOWER
I20260430 20:01:39.987823 12703 consensus_queue.cc:260] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:39.988198 12703 ts_tablet_manager.cc:1434] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:01:39.988502 12751 ts_tablet_manager.cc:1916] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:39.992362 12751 ts_tablet_manager.cc:1929] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:39.992565 12751 log.cc:1199] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/7818fb0e0cd647f2a8ce023822eff1da
I20260430 20:01:39.992978 12751 ts_tablet_manager.cc:1950] T 7818fb0e0cd647f2a8ce023822eff1da P 7679f60c915e4d1bb9d772235dd3608f: Deleting consensus metadata
I20260430 20:01:39.994410 12178 catalog_manager.cc:5002] TS 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917): tablet 7818fb0e0cd647f2a8ce023822eff1da (table test-table [id=361e94792fc548438569adaf1da58723]) successfully deleted
I20260430 20:01:40.170600 12726 raft_consensus.cc:493] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:40.170782 12726 raft_consensus.cc:515] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:40.171346 12726 leader_election.cc:290] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:40.171856 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6019a52c35f54d03b74e8af9941cb08e" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6" is_pre_election: true
I20260430 20:01:40.171831 12349 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6019a52c35f54d03b74e8af9941cb08e" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f" is_pre_election: true
I20260430 20:01:40.172012 12634 raft_consensus.cc:2468] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 0.
I20260430 20:01:40.172012 12349 raft_consensus.cc:2468] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 0.
I20260430 20:01:40.172330 12426 leader_election.cc:304] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1fb83251ca0481a8668660e3299c3a6, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:40.172603 12726 raft_consensus.cc:2804] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:40.172678 12726 raft_consensus.cc:493] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:40.172727 12726 raft_consensus.cc:3060] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:40.174176 12726 raft_consensus.cc:515] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:40.174641 12726 leader_election.cc:290] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 election: Requested vote from peers 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:40.175041 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6019a52c35f54d03b74e8af9941cb08e" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6"
I20260430 20:01:40.175063 12349 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6019a52c35f54d03b74e8af9941cb08e" candidate_uuid: "f636f078b88a4cffb379cc128892904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679f60c915e4d1bb9d772235dd3608f"
I20260430 20:01:40.175176 12634 raft_consensus.cc:3060] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:40.175237 12349 raft_consensus.cc:3060] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:40.176170 12634 raft_consensus.cc:2468] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 1.
I20260430 20:01:40.176304 12349 raft_consensus.cc:2468] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f636f078b88a4cffb379cc128892904a in term 1.
I20260430 20:01:40.176570 12426 leader_election.cc:304] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1fb83251ca0481a8668660e3299c3a6, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:40.176785 12726 raft_consensus.cc:2804] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:40.176892 12726 raft_consensus.cc:697] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 1 LEADER]: Becoming Leader. State: Replica: f636f078b88a4cffb379cc128892904a, State: Running, Role: LEADER
I20260430 20:01:40.177182 12726 consensus_queue.cc:237] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:40.179176 12190 catalog_manager.cc:5671] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a reported cstate change: term changed from 0 to 1, leader changed from <none> to f636f078b88a4cffb379cc128892904a (127.10.46.2). New cstate: current_term: 1 leader_uuid: "f636f078b88a4cffb379cc128892904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:40.231419 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:40.228509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.10.46.2:35061)
0430 20:01:40.228690 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:40.228694 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:40.228707 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:01:40.228745 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:40.228747 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:40.228793 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:40.228862 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:40.229327 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.229804 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:40.230648 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.230784 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:40.230818 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:40.230886 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:40.230950 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:40.231121 (+ 171us) server_negotiation.cc:300] Negotiation successful
0430 20:01:40.231243 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 20:01:40.240461 12746 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:40.237484 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.10.46.2:59513)
0430 20:01:40.237636 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:40.237651 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:40.237734 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:40.238005 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:40.238008 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:40.238018 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:40.238250 (+ 232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:40.238256 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.239003 (+ 747us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:40.239006 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:40.239619 (+ 613us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:40.239626 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.239738 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:40.239760 (+ 22us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:40.240146 (+ 386us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:40.240151 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 20:01:40.240214 (+ 63us) client_negotiation.cc:241] Negotiation successful
0430 20:01:40.240277 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260430 20:01:40.240491 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:40.237568 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:59513 (local address 127.10.46.2:35061)
0430 20:01:40.237672 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:40.237676 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:40.237690 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:01:40.237860 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:40.237863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:40.237908 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:40.237979 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:40.238376 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.238876 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:40.239741 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:40.239952 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:40.239991 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:40.240061 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:40.240129 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:40.240231 (+ 102us) server_negotiation.cc:300] Negotiation successful
0430 20:01:40.240272 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260430 20:01:40.246135 12634 raft_consensus.cc:1275] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Refusing update from remote peer f636f078b88a4cffb379cc128892904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:40.246496 12350 raft_consensus.cc:1275] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Refusing update from remote peer f636f078b88a4cffb379cc128892904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:40.246524 12726 consensus_queue.cc:1048] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:40.246887 12726 consensus_queue.cc:1048] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:40.252267 12732 mvcc.cc:204] Tried to move back new op lower bound from 7280964813805248512 to 7280964813527633920. Current Snapshot: MvccSnapshot[applied={T|T < 7280964813805248512}]
W20260430 20:01:40.263707 12731 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260430 20:01:41.844808 12190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:41.844993 12190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:41.848052 12190 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Sending DeleteTablet for 3 replicas of tablet 6019a52c35f54d03b74e8af9941cb08e
I20260430 20:01:41.849651 12472 tablet_service.cc:1558] Processing DeleteTablet for tablet 6019a52c35f54d03b74e8af9941cb08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:41 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260430 20:01:41.849651 12330 tablet_service.cc:1558] Processing DeleteTablet for tablet 6019a52c35f54d03b74e8af9941cb08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:41 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:55020
I20260430 20:01:41.849741 12614 tablet_service.cc:1558] Processing DeleteTablet for tablet 6019a52c35f54d03b74e8af9941cb08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:41 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:47918
I20260430 20:01:41.850113 12781 tablet_replica.cc:333] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: stopping tablet replica
I20260430 20:01:41.850261 12781 raft_consensus.cc:2243] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:01:41.850447 12780 tablet_replica.cc:333] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: stopping tablet replica
I20260430 20:01:41.850620 12780 raft_consensus.cc:2243] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:41.850718 12781 raft_consensus.cc:2272] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:41.850800 12780 raft_consensus.cc:2272] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:41.852373 12780 ts_tablet_manager.cc:1916] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:41.853848 12781 ts_tablet_manager.cc:1916] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:41.854087 12782 tablet_replica.cc:333] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: stopping tablet replica
I20260430 20:01:41.854554 12782 raft_consensus.cc:2243] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:41.854779 12782 raft_consensus.cc:2272] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:41.855646 12782 ts_tablet_manager.cc:1916] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
Apr 30 20:01:41 dist-test-slave-0bdw krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260430 20:01:41.923065 12781 ts_tablet_manager.cc:1929] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 20:01:41.923331 12781 log.cc:1199] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/6019a52c35f54d03b74e8af9941cb08e
I20260430 20:01:41.923722 12781 ts_tablet_manager.cc:1950] T 6019a52c35f54d03b74e8af9941cb08e P f636f078b88a4cffb379cc128892904a: Deleting consensus metadata
I20260430 20:01:41.927119 12780 ts_tablet_manager.cc:1929] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 20:01:41.927136 12782 ts_tablet_manager.cc:1929] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 20:01:41.927245 12782 log.cc:1199] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/6019a52c35f54d03b74e8af9941cb08e
I20260430 20:01:41.927374 12780 log.cc:1199] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/6019a52c35f54d03b74e8af9941cb08e
I20260430 20:01:41.927634 12782 ts_tablet_manager.cc:1950] T 6019a52c35f54d03b74e8af9941cb08e P f1fb83251ca0481a8668660e3299c3a6: Deleting consensus metadata
I20260430 20:01:41.927806 12780 ts_tablet_manager.cc:1950] T 6019a52c35f54d03b74e8af9941cb08e P 7679f60c915e4d1bb9d772235dd3608f: Deleting consensus metadata
Apr 30 20:01:41 dist-test-slave-0bdw krb5kdc[12143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:41.937597 12790 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:41.924316 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.10.46.62:35489)
0430 20:01:41.924652 (+ 336us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:41.924656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:41.924672 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:01:41.924811 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:41.924822 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:41.924879 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:41.924988 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:41.925842 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:41.926465 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:41.927253 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:41.927478 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:41.934010 (+ 6532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:41.934037 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:41.934040 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:41.934079 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:41.935678 (+ 1599us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:41.936255 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:41.936258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:41.936259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:41.936316 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:41.936691 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:41.936695 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:41.936696 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:41.936866 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:41.936987 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:41.937249 (+ 262us) server_negotiation.cc:300] Negotiation successful
0430 20:01:41.937365 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260430 20:01:41.940807 12190 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:01:41.941304 12190 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:41.947278 12330 tablet_service.cc:1511] Processing CreateTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 (DEFAULT_TABLE table=test-table [id=f13434b43fea46d88f5a796ca6ab801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:41.947633 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f90014a9c964f9e9a5197eaabfa2d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:41.947575 12614 tablet_service.cc:1511] Processing CreateTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 (DEFAULT_TABLE table=test-table [id=f13434b43fea46d88f5a796ca6ab801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:41.947885 12614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f90014a9c964f9e9a5197eaabfa2d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:41.950150 12791 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Bootstrap starting.
I20260430 20:01:41.950388 12472 tablet_service.cc:1511] Processing CreateTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 (DEFAULT_TABLE table=test-table [id=f13434b43fea46d88f5a796ca6ab801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:41.950702 12472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f90014a9c964f9e9a5197eaabfa2d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:41.951458 12791 tablet_bootstrap.cc:654] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:41.951650 12792 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Bootstrap starting.
I20260430 20:01:41.952653 12792 tablet_bootstrap.cc:654] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:41.953207 12794 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Bootstrap starting.
I20260430 20:01:41.954104 12794 tablet_bootstrap.cc:654] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:41.954522 12792 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: No bootstrap required, opened a new log
I20260430 20:01:41.954653 12792 ts_tablet_manager.cc:1403] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:41.955727 12791 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: No bootstrap required, opened a new log
I20260430 20:01:41.955859 12791 ts_tablet_manager.cc:1403] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260430 20:01:41.956164 12792 raft_consensus.cc:359] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.956420 12792 raft_consensus.cc:385] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:41.956490 12792 raft_consensus.cc:740] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Initialized, Role: FOLLOWER
I20260430 20:01:41.956470 12791 raft_consensus.cc:359] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.956615 12791 raft_consensus.cc:385] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:41.956653 12791 raft_consensus.cc:740] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Initialized, Role: FOLLOWER
I20260430 20:01:41.956715 12794 tablet_bootstrap.cc:492] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: No bootstrap required, opened a new log
I20260430 20:01:41.956708 12792 consensus_queue.cc:260] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.956799 12794 ts_tablet_manager.cc:1403] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 20:01:41.956792 12791 consensus_queue.cc:260] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.957235 12791 ts_tablet_manager.cc:1434] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260430 20:01:41.957252 12792 ts_tablet_manager.cc:1434] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:41.957280 12794 raft_consensus.cc:359] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.957923 12794 raft_consensus.cc:385] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:41.958007 12794 raft_consensus.cc:740] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636f078b88a4cffb379cc128892904a, State: Initialized, Role: FOLLOWER
I20260430 20:01:41.958489 12794 consensus_queue.cc:260] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:41.958927 12794 ts_tablet_manager.cc:1434] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 20:01:41.967638 12178 catalog_manager.cc:5002] TS f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287): tablet 6019a52c35f54d03b74e8af9941cb08e (table test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]) successfully deleted
I20260430 20:01:41.968039 12178 catalog_manager.cc:5002] TS 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917): tablet 6019a52c35f54d03b74e8af9941cb08e (table test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]) successfully deleted
I20260430 20:01:41.971410 12178 catalog_manager.cc:5002] TS f636f078b88a4cffb379cc128892904a (127.10.46.2:35061): tablet 6019a52c35f54d03b74e8af9941cb08e (table test-table [id=6dfbbd4cc2744ceaa5e5171f96e62253]) successfully deleted
I20260430 20:01:42.213270 12798 raft_consensus.cc:493] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:42.213455 12798 raft_consensus.cc:515] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.214735 12798 leader_election.cc:290] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f636f078b88a4cffb379cc128892904a (127.10.46.2:35061), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:42.218652 12799 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.214972 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.10.46.1:50105)
0430 20:01:42.215473 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:42.215495 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:42.215627 (+ 132us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:42.215969 (+ 342us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:42.215998 (+ 29us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:42.216020 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:42.216328 (+ 308us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.216336 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.217459 (+ 1123us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.217464 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:42.218302 (+ 838us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.218311 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.218420 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.218437 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 20:01:42.218470 (+ 33us) client_negotiation.cc:241] Negotiation successful
0430 20:01:42.218515 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":186,"threads_started":1}
I20260430 20:01:42.219190 12801 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.215486 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:43287 (local address 127.10.46.1:52083)
0430 20:01:42.215801 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:42.215815 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:42.215919 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:42.216343 (+ 424us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:42.216348 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:42.216441 (+ 93us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:01:42.216826 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.216834 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.217996 (+ 1162us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.218000 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:42.218785 (+ 785us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.218795 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.218927 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.218943 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:01:42.218993 (+ 50us) client_negotiation.cc:241] Negotiation successful
0430 20:01:42.219054 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260430 20:01:42.219652 12800 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.215255 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50105 (local address 127.10.46.2:35061)
0430 20:01:42.215596 (+ 341us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:42.215601 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:42.215618 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:01:42.215748 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:42.215752 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:42.215847 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:42.215938 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:42.216499 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.217301 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.218524 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.219406 (+ 882us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.219461 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:42.219507 (+ 46us) server_negotiation.cc:300] Negotiation successful
0430 20:01:42.219554 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":114,"threads_started":1}
I20260430 20:01:42.219851 12802 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.215656 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:52083 (local address 127.10.46.3:43287)
0430 20:01:42.216000 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:42.216004 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:42.216017 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:01:42.216067 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:42.216070 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:42.216122 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:42.216232 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:01:42.217073 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.217781 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.219033 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.219526 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.219567 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:42.219612 (+ 45us) server_negotiation.cc:300] Negotiation successful
0430 20:01:42.219657 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":156,"threads_started":1}
I20260430 20:01:42.220157 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f90014a9c964f9e9a5197eaabfa2d33" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a" is_pre_election: true
I20260430 20:01:42.220320 12492 raft_consensus.cc:2468] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 0.
I20260430 20:01:42.220414 12633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f90014a9c964f9e9a5197eaabfa2d33" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6" is_pre_election: true
I20260430 20:01:42.220590 12633 raft_consensus.cc:2468] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 0.
I20260430 20:01:42.220889 12284 leader_election.cc:304] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:42.221417 12798 raft_consensus.cc:2804] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:42.221547 12798 raft_consensus.cc:493] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:42.221577 12798 raft_consensus.cc:3060] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.285604 12798 raft_consensus.cc:515] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.286175 12798 leader_election.cc:290] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 election: Requested vote from peers f636f078b88a4cffb379cc128892904a (127.10.46.2:35061), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:42.286775 12633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f90014a9c964f9e9a5197eaabfa2d33" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6"
I20260430 20:01:42.286801 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f90014a9c964f9e9a5197eaabfa2d33" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a"
I20260430 20:01:42.286936 12633 raft_consensus.cc:3060] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.286986 12492 raft_consensus.cc:3060] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.288591 12633 raft_consensus.cc:2468] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 1.
I20260430 20:01:42.288707 12492 raft_consensus.cc:2468] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 1.
I20260430 20:01:42.289227 12284 leader_election.cc:304] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:42.289552 12798 raft_consensus.cc:2804] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:42.289885 12798 raft_consensus.cc:697] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 1 LEADER]: Becoming Leader. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Running, Role: LEADER
I20260430 20:01:42.290225 12798 consensus_queue.cc:237] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.294226 12190 catalog_manager.cc:5671] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1). New cstate: current_term: 1 leader_uuid: "7679f60c915e4d1bb9d772235dd3608f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:42.343149 12814 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.338522 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.10.46.1:34917)
0430 20:01:42.338994 (+ 472us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:42.338998 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:42.339015 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:01:42.339158 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:42.339162 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:42.339236 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:42.339416 (+ 180us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:42.339984 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.340707 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.341701 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.341927 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.342109 (+ 182us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:42.342595 (+ 486us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:42.342734 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:42.342856 (+ 122us) server_negotiation.cc:300] Negotiation successful
0430 20:01:42.342918 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":147,"threads_started":1}
I20260430 20:01:42.356240 12800 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.352695 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:40381 (local address 127.10.46.2:35061)
0430 20:01:42.352897 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:42.352902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:42.352921 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:01:42.353225 (+ 304us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:42.353231 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:42.353300 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:42.353399 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:42.353817 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.354323 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.355349 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.355567 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.355607 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:42.355696 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:42.355848 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:42.356017 (+ 169us) server_negotiation.cc:300] Negotiation successful
0430 20:01:42.356066 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:01:42.356256 12817 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.352235 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.10.46.1:40381)
0430 20:01:42.352728 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:42.352771 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:42.352959 (+ 188us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:42.353417 (+ 458us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:42.353420 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:42.353430 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:42.353698 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.353703 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.354445 (+ 742us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.354449 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:42.355197 (+ 748us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.355207 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.355379 (+ 172us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.355436 (+ 57us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:42.355822 (+ 386us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:42.355829 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:01:42.356003 (+ 174us) client_negotiation.cc:241] Negotiation successful
0430 20:01:42.356094 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":112,"threads_started":1}
I20260430 20:01:42.365047 12492 raft_consensus.cc:1275] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Refusing update from remote peer 7679f60c915e4d1bb9d772235dd3608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:42.365166 12633 raft_consensus.cc:1275] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Refusing update from remote peer 7679f60c915e4d1bb9d772235dd3608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:42.366039 12798 consensus_queue.cc:1048] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:42.366334 12803 consensus_queue.cc:1048] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:42.397104 12823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.389158 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.10.46.2:49581)
0430 20:01:42.391952 (+ 2794us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:42.391971 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:42.392110 (+ 139us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:42.392519 (+ 409us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:42.392524 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:42.392535 (+ 11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:01:42.392923 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.392933 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.394598 (+ 1665us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.394603 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:42.395482 (+ 879us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:42.395496 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.395646 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.395678 (+ 32us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:01:42.396590 (+ 912us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:01:42.396598 (+ 8us) client_negotiation.cc:770] Sending connection context
0430 20:01:42.396690 (+ 92us) client_negotiation.cc:241] Negotiation successful
0430 20:01:42.396771 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2586,"thread_start_us":200,"threads_started":1}
I20260430 20:01:42.397117 12814 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:42.389262 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:49581 (local address 127.10.46.1:34917)
0430 20:01:42.389409 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:42.389413 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:42.392217 (+ 2804us) server_negotiation.cc:408] Connection header received
0430 20:01:42.392292 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:42.392296 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:42.392383 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:42.392462 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:01:42.393148 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.394216 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:42.395681 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:42.396039 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:42.396104 (+ 65us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:01:42.396355 (+ 251us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:01:42.396539 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:42.396745 (+ 206us) server_negotiation.cc:300] Negotiation successful
0430 20:01:42.396860 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 20:01:42.431608 12190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:42.431841 12190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:42.434728 12190 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Sending DeleteTablet for 3 replicas of tablet 5f90014a9c964f9e9a5197eaabfa2d33
I20260430 20:01:42.435870 12472 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:42 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260430 20:01:42.435861 12330 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:42 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:55020
I20260430 20:01:42.436062 12614 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f90014a9c964f9e9a5197eaabfa2d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:42 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:47918
I20260430 20:01:42.436123 12781 tablet_replica.cc:333] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: stopping tablet replica
I20260430 20:01:42.436141 12780 tablet_replica.cc:333] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: stopping tablet replica
I20260430 20:01:42.436244 12782 tablet_replica.cc:333] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: stopping tablet replica
I20260430 20:01:42.436254 12780 raft_consensus.cc:2243] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:01:42.436241 12781 raft_consensus.cc:2243] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:42.436343 12782 raft_consensus.cc:2243] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:42.436393 12781 raft_consensus.cc:2272] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:42.436453 12782 raft_consensus.cc:2272] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:42.436568 12780 raft_consensus.cc:2272] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:42.437281 12781 ts_tablet_manager.cc:1916] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:42.437299 12780 ts_tablet_manager.cc:1916] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:42.438447 12782 ts_tablet_manager.cc:1916] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:42.439113 12190 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:01:42.439656 12190 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:01:42.445125 12330 tablet_service.cc:1511] Processing CreateTablet for tablet 0fcef054aee14b17be63a673c31e739c (DEFAULT_TABLE table=test-table [id=3032500402384eaca87ab234dff1065b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:42.445135 12472 tablet_service.cc:1511] Processing CreateTablet for tablet 0fcef054aee14b17be63a673c31e739c (DEFAULT_TABLE table=test-table [id=3032500402384eaca87ab234dff1065b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:42.445605 12614 tablet_service.cc:1511] Processing CreateTablet for tablet 0fcef054aee14b17be63a673c31e739c (DEFAULT_TABLE table=test-table [id=3032500402384eaca87ab234dff1065b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:01:42.487052 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcef054aee14b17be63a673c31e739c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:42.491324 12614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcef054aee14b17be63a673c31e739c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:42.493283 12827 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f: Bootstrap starting.
I20260430 20:01:42.494067 12472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcef054aee14b17be63a673c31e739c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:42.494414 12828 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Bootstrap starting.
I20260430 20:01:42.494478 12827 tablet_bootstrap.cc:654] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:42.495415 12828 tablet_bootstrap.cc:654] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:42.495801 12827 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f: No bootstrap required, opened a new log
I20260430 20:01:42.495914 12827 ts_tablet_manager.cc:1403] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:42.496455 12827 raft_consensus.cc:359] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.496577 12827 raft_consensus.cc:385] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:42.496606 12831 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: Bootstrap starting.
I20260430 20:01:42.496618 12827 raft_consensus.cc:740] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Initialized, Role: FOLLOWER
I20260430 20:01:42.496749 12827 consensus_queue.cc:260] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.497041 12827 ts_tablet_manager.cc:1434] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:01:42.499682 12831 tablet_bootstrap.cc:654] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:42.500226 12828 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: No bootstrap required, opened a new log
I20260430 20:01:42.500474 12828 ts_tablet_manager.cc:1403] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260430 20:01:42.501135 12828 raft_consensus.cc:359] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.501421 12828 raft_consensus.cc:385] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:42.501480 12831 tablet_bootstrap.cc:492] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: No bootstrap required, opened a new log
I20260430 20:01:42.501510 12828 raft_consensus.cc:740] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb83251ca0481a8668660e3299c3a6, State: Initialized, Role: FOLLOWER
I20260430 20:01:42.501587 12831 ts_tablet_manager.cc:1403] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260430 20:01:42.501803 12828 consensus_queue.cc:260] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.502048 12831 raft_consensus.cc:359] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.502164 12831 raft_consensus.cc:385] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:42.502219 12831 raft_consensus.cc:740] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636f078b88a4cffb379cc128892904a, State: Initialized, Role: FOLLOWER
I20260430 20:01:42.502357 12828 ts_tablet_manager.cc:1434] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 20:01:42.502363 12831 consensus_queue.cc:260] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.502665 12831 ts_tablet_manager.cc:1434] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:01:42.584779 12780 ts_tablet_manager.cc:1929] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:42.584779 12782 ts_tablet_manager.cc:1929] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:42.584908 12780 log.cc:1199] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/5f90014a9c964f9e9a5197eaabfa2d33
I20260430 20:01:42.584906 12782 log.cc:1199] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/5f90014a9c964f9e9a5197eaabfa2d33
I20260430 20:01:42.585225 12780 ts_tablet_manager.cc:1950] T 5f90014a9c964f9e9a5197eaabfa2d33 P 7679f60c915e4d1bb9d772235dd3608f: Deleting consensus metadata
I20260430 20:01:42.585229 12782 ts_tablet_manager.cc:1950] T 5f90014a9c964f9e9a5197eaabfa2d33 P f1fb83251ca0481a8668660e3299c3a6: Deleting consensus metadata
I20260430 20:01:42.585566 12781 ts_tablet_manager.cc:1929] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:01:42.585659 12781 log.cc:1199] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/5f90014a9c964f9e9a5197eaabfa2d33
I20260430 20:01:42.585987 12781 ts_tablet_manager.cc:1950] T 5f90014a9c964f9e9a5197eaabfa2d33 P f636f078b88a4cffb379cc128892904a: Deleting consensus metadata
I20260430 20:01:42.592536 12178 catalog_manager.cc:5002] TS 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917): tablet 5f90014a9c964f9e9a5197eaabfa2d33 (table test-table [id=f13434b43fea46d88f5a796ca6ab801e]) successfully deleted
I20260430 20:01:42.592883 12178 catalog_manager.cc:5002] TS f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287): tablet 5f90014a9c964f9e9a5197eaabfa2d33 (table test-table [id=f13434b43fea46d88f5a796ca6ab801e]) successfully deleted
I20260430 20:01:42.607038 12178 catalog_manager.cc:5002] TS f636f078b88a4cffb379cc128892904a (127.10.46.2:35061): tablet 5f90014a9c964f9e9a5197eaabfa2d33 (table test-table [id=f13434b43fea46d88f5a796ca6ab801e]) successfully deleted
I20260430 20:01:42.620273 12825 raft_consensus.cc:493] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:01:42.620435 12825 raft_consensus.cc:515] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.622170 12825 leader_election.cc:290] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f636f078b88a4cffb379cc128892904a (127.10.46.2:35061), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:42.622627 12633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fcef054aee14b17be63a673c31e739c" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6" is_pre_election: true
I20260430 20:01:42.622627 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fcef054aee14b17be63a673c31e739c" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a" is_pre_election: true
I20260430 20:01:42.622766 12492 raft_consensus.cc:2468] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 0.
I20260430 20:01:42.622766 12633 raft_consensus.cc:2468] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 0.
I20260430 20:01:42.623157 12284 leader_election.cc:304] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:42.623461 12825 raft_consensus.cc:2804] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:01:42.623528 12825 raft_consensus.cc:493] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:01:42.623571 12825 raft_consensus.cc:3060] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.683372 12825 raft_consensus.cc:515] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.683967 12825 leader_election.cc:290] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 election: Requested vote from peers f636f078b88a4cffb379cc128892904a (127.10.46.2:35061), f1fb83251ca0481a8668660e3299c3a6 (127.10.46.3:43287)
I20260430 20:01:42.684545 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fcef054aee14b17be63a673c31e739c" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636f078b88a4cffb379cc128892904a"
I20260430 20:01:42.684576 12633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fcef054aee14b17be63a673c31e739c" candidate_uuid: "7679f60c915e4d1bb9d772235dd3608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb83251ca0481a8668660e3299c3a6"
I20260430 20:01:42.684700 12492 raft_consensus.cc:3060] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.684747 12633 raft_consensus.cc:3060] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:42.686038 12492 raft_consensus.cc:2468] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 1.
I20260430 20:01:42.686301 12633 raft_consensus.cc:2468] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7679f60c915e4d1bb9d772235dd3608f in term 1.
I20260430 20:01:42.686509 12284 leader_election.cc:304] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7679f60c915e4d1bb9d772235dd3608f, f636f078b88a4cffb379cc128892904a; no voters:
I20260430 20:01:42.686758 12825 raft_consensus.cc:2804] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:42.686872 12825 raft_consensus.cc:697] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [term 1 LEADER]: Becoming Leader. State: Replica: 7679f60c915e4d1bb9d772235dd3608f, State: Running, Role: LEADER
I20260430 20:01:42.687048 12825 consensus_queue.cc:237] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } }
I20260430 20:01:42.690212 12191 catalog_manager.cc:5671] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1). New cstate: current_term: 1 leader_uuid: "7679f60c915e4d1bb9d772235dd3608f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679f60c915e4d1bb9d772235dd3608f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 34917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 } health_report { overall_health: UNKNOWN } } }
I20260430 20:01:42.766106 12633 raft_consensus.cc:1275] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Refusing update from remote peer 7679f60c915e4d1bb9d772235dd3608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:42.766106 12492 raft_consensus.cc:1275] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Refusing update from remote peer 7679f60c915e4d1bb9d772235dd3608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:01:42.766527 12825 consensus_queue.cc:1048] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f636f078b88a4cffb379cc128892904a" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:01:42.766710 12803 consensus_queue.cc:1048] T 0fcef054aee14b17be63a673c31e739c P 7679f60c915e4d1bb9d772235dd3608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fb83251ca0481a8668660e3299c3a6" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 20:01:42.785292 12808 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260430 20:01:44.363976 12190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:44.364226 12190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:01:44.367704 12190 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 94f6b46ea13c4794aaccdf169d3737fc: Sending DeleteTablet for 3 replicas of tablet 0fcef054aee14b17be63a673c31e739c
I20260430 20:01:44.368541 12472 tablet_service.cc:1558] Processing DeleteTablet for tablet 0fcef054aee14b17be63a673c31e739c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:44 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260430 20:01:44.368541 12330 tablet_service.cc:1558] Processing DeleteTablet for tablet 0fcef054aee14b17be63a673c31e739c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:44 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:55020
I20260430 20:01:44.368616 12614 tablet_service.cc:1558] Processing DeleteTablet for tablet 0fcef054aee14b17be63a673c31e739c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:01:44 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:47918
I20260430 20:01:44.369100 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12265
I20260430 20:01:44.369118 12857 tablet_replica.cc:333] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: stopping tablet replica
I20260430 20:01:44.371862 12857 raft_consensus.cc:2243] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:44.372167 12857 raft_consensus.cc:2272] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:44.372177 12856 tablet_replica.cc:333] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: stopping tablet replica
I20260430 20:01:44.372316 12856 raft_consensus.cc:2243] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:01:44.372478 12856 raft_consensus.cc:2272] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:01:44.373263 12857 ts_tablet_manager.cc:1916] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:01:44.373263 12856 ts_tablet_manager.cc:1916] T 0fcef054aee14b17be63a673c31e739c P f636f078b88a4cffb379cc128892904a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260430 20:01:44.380895 12178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.46.1:34917 (error 108)
I20260430 20:01:44.381145 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12407
I20260430 20:01:44.383524 12859 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:44.383018 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.0.0.1:55040)
0430 20:01:44.383336 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:44.383401 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:34917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
W20260430 20:01:44.383800 12178 catalog_manager.cc:4729] TS 7679f60c915e4d1bb9d772235dd3608f (127.10.46.1:34917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fcef054aee14b17be63a673c31e739c: Network error: Client connection negotiation failed: client connection to 127.10.46.1:34917: connect: Connection refused (error 111)
I20260430 20:01:44.437552 12859 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:44.437166 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.0.0.1:55042)
0430 20:01:44.437393 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:44.437455 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:34917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260430 20:01:44.438742 12859 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:44.438383 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:34917 (local address 127.0.0.1:55048)
0430 20:01:44.438584 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:44.438641 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:34917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260430 20:01:44.445637 12857 ts_tablet_manager.cc:1929] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 20:01:44.445751 12857 log.cc:1199] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/0fcef054aee14b17be63a673c31e739c
I20260430 20:01:44.445937 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12549
I20260430 20:01:44.446141 12857 ts_tablet_manager.cc:1950] T 0fcef054aee14b17be63a673c31e739c P f1fb83251ca0481a8668660e3299c3a6: Deleting consensus metadata
I20260430 20:01:44.447995 12859 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:44.447630 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:35061 (local address 127.0.0.1:43350)
0430 20:01:44.447801 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:44.447869 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.2:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260430 20:01:44.477243 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12159
2026-04-30T20:01:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8323 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:45 dist-test-slave-0bdw krb5kdc[12862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:45 dist-test-slave-0bdw krb5kdc[12862](info): set up 2 sockets
Apr 30 20:01:45 dist-test-slave-0bdw krb5kdc[12862](info): commencing operation
krb5kdc: starting...
W20260430 20:01:47.217334 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.712s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:47 dist-test-slave-0bdw krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:47.401849 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34973
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:47.527987 12878 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:47.528301 12878 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:47.528402 12878 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:47.532960 12878 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:47.533103 12878 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:47.533152 12878 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:47.533214 12878 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:47.533277 12878 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:47.538580 12878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:34973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:47.539911 12878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:47.540827 12878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:47.547399 12883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:47.547399 12886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:47.547607 12884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:47.547828 12878 server_base.cc:1061] running on GCE node
I20260430 20:01:47.548362 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:47.549269 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:47.550477 12878 hybrid_clock.cc:648] HybridClock initialized: now 1777579307550450 us; error 42 us; skew 500 ppm
Apr 30 20:01:47 dist-test-slave-0bdw krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:47.553392 12878 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:47.554512 12878 webserver.cc:492] Webserver started at http://127.10.46.62:46097/ using document root <none> and password file <none>
I20260430 20:01:47.555172 12878 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:47.555223 12878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:47.555441 12878 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:47.557328 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "c1b46928e36f4bbcbe493668f6838bb2"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "45812ad6fce69e7c4b54bdb2776d2a99"
server_key_iv: "f4d4087864297ed9c8f08975f38bf300"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.557863 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "c1b46928e36f4bbcbe493668f6838bb2"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "45812ad6fce69e7c4b54bdb2776d2a99"
server_key_iv: "f4d4087864297ed9c8f08975f38bf300"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.561771 12878 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:01:47.564528 12893 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:47.565840 12878 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:47.566025 12878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "c1b46928e36f4bbcbe493668f6838bb2"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "45812ad6fce69e7c4b54bdb2776d2a99"
server_key_iv: "f4d4087864297ed9c8f08975f38bf300"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.566157 12878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:47.627848 12878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:47.636289 12878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:47.636798 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:47.647581 12878 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:34973
I20260430 20:01:47.647579 12945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:34973 every 8 connection(s)
I20260430 20:01:47.649387 12878 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:47.650875 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12878
I20260430 20:01:47.651073 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:47.651533 10424 external_mini_cluster.cc:1468] Setting key 6fab00fcd6ccb456617e97985d4700b3
I20260430 20:01:47.654647 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:47.662302 12946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2: Bootstrap starting.
Apr 30 20:01:47 dist-test-slave-0bdw krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:47.664958 12946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:47.665789 12946 log.cc:826] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:47.669170 12946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2: No bootstrap required, opened a new log
I20260430 20:01:47.670075 12949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:47.653749 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.10.46.62:34973)
0430 20:01:47.654485 (+ 736us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:47.654501 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:47.654783 (+ 282us) server_negotiation.cc:408] Connection header received
0430 20:01:47.657038 (+ 2255us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:47.657070 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:47.657429 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:47.657863 (+ 434us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:47.658869 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:47.659642 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:47.660517 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:47.660843 (+ 326us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:47.663305 (+ 2462us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:47.663368 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:47.663383 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:47.663419 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:47.665473 (+ 2054us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:47.666674 (+ 1201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:47.666679 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:47.666685 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:47.666773 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:47.667262 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:47.667266 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:47.667270 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:47.667704 (+ 434us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:47.667980 (+ 276us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:47.668794 (+ 814us) server_negotiation.cc:300] Negotiation successful
0430 20:01:47.669247 (+ 453us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":169,"threads_started":1}
I20260430 20:01:47.674525 12946 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } }
I20260430 20:01:47.675966 12946 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:47.676054 12946 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b46928e36f4bbcbe493668f6838bb2, State: Initialized, Role: FOLLOWER
I20260430 20:01:47.676587 12946 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } }
I20260430 20:01:47.676716 12946 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:47.676759 12946 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:47.676843 12946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:47.678318 12946 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } }
I20260430 20:01:47.678802 12946 leader_election.cc:304] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1b46928e36f4bbcbe493668f6838bb2; no voters:
I20260430 20:01:47.679342 12946 leader_election.cc:290] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:47.679476 12951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:47.679703 12951 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [term 1 LEADER]: Becoming Leader. State: Replica: c1b46928e36f4bbcbe493668f6838bb2, State: Running, Role: LEADER
I20260430 20:01:47.680096 12951 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } }
I20260430 20:01:47.680694 12946 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:47.681895 12952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b46928e36f4bbcbe493668f6838bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } } }
I20260430 20:01:47.682051 12952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:47.681896 12953 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b46928e36f4bbcbe493668f6838bb2. Latest consensus state: current_term: 1 leader_uuid: "c1b46928e36f4bbcbe493668f6838bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b46928e36f4bbcbe493668f6838bb2" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34973 } } }
I20260430 20:01:47.682106 12953 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:47.683447 12960 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:47.686720 12960 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:47.693574 12960 catalog_manager.cc:1357] Generated new cluster ID: f0bee2bc57004001b4a53fcae533ea91
I20260430 20:01:47.693673 12960 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:47.726541 12960 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:47.727881 12960 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:47.736523 12960 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c1b46928e36f4bbcbe493668f6838bb2: Generated new TSK 0
I20260430 20:01:47.737398 12960 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:47.821046 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34973
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:47.953421 12974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:47.953698 12974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:47.953805 12974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:47.958050 12974 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:47.958137 12974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:47.958223 12974 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:47.964151 12974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:47.965718 12974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:47.966722 12974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:47.975186 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:47.975173 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:47.975288 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:47.975934 12974 server_base.cc:1061] running on GCE node
I20260430 20:01:47.976706 12974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:47.977427 12974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:47.978740 12974 hybrid_clock.cc:648] HybridClock initialized: now 1777579307978690 us; error 64 us; skew 500 ppm
Apr 30 20:01:47 dist-test-slave-0bdw krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:47.982010 12974 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:47.983179 12974 webserver.cc:492] Webserver started at http://127.10.46.1:34981/ using document root <none> and password file <none>
I20260430 20:01:47.983760 12974 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:47.983886 12974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:47.984118 12974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:47.986095 12974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "560d8fc92927495da4839e7392ebba32"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "745d798f7653770563038505d1261c34"
server_key_iv: "38728cdb66044887f1dc4f37d4d277e6"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.986665 12974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "560d8fc92927495da4839e7392ebba32"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "745d798f7653770563038505d1261c34"
server_key_iv: "38728cdb66044887f1dc4f37d4d277e6"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.990563 12974 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260430 20:01:47.993587 12989 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:47.995251 12974 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 20:01:47.995461 12974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "560d8fc92927495da4839e7392ebba32"
format_stamp: "Formatted at 2026-04-30 20:01:47 on dist-test-slave-0bdw"
server_key: "745d798f7653770563038505d1261c34"
server_key_iv: "38728cdb66044887f1dc4f37d4d277e6"
server_key_version: "encryptionkey@0"
I20260430 20:01:47.995587 12974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:48.038998 12974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:48.043222 12974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:48.043574 12974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:48.044292 12974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:48.045307 12974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:48.045382 12974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.045450 12974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:48.045496 12974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.056118 12974 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:33893
I20260430 20:01:48.056285 13102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:33893 every 8 connection(s)
I20260430 20:01:48.057235 12974 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:48.059235 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 12974
I20260430 20:01:48.059347 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:48.059623 10424 external_mini_cluster.cc:1468] Setting key 5e7753a55c795d2f4929af2ffb0c361e
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:01:48.071892 12949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.059269 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50973 (local address 127.10.46.62:34973)
0430 20:01:48.059424 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:48.059428 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:48.060387 (+ 959us) server_negotiation.cc:408] Connection header received
0430 20:01:48.061504 (+ 1117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:48.061508 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:48.061563 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:48.061647 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:48.063289 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.063945 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.064732 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.064915 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.067480 (+ 2565us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:48.067502 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:48.067504 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:48.067531 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:48.069088 (+ 1557us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.069687 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.069690 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.069692 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.069742 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.070151 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.070154 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.070155 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.070311 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:48.070406 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:48.071307 (+ 901us) server_negotiation.cc:300] Negotiation successful
0430 20:01:48.071542 (+ 235us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:01:48.072129 13105 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.059592 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34973 (local address 127.10.46.1:50973)
0430 20:01:48.060242 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:48.060295 (+ 53us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:48.061310 (+ 1015us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:48.061832 (+ 522us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:48.061840 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:48.062238 (+ 398us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:48.063117 (+ 879us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.063130 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.064077 (+ 947us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.064083 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:48.064598 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.064606 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.064851 (+ 245us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.065503 (+ 652us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:48.065523 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:48.067291 (+ 1768us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:48.069231 (+ 1940us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.069238 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.069253 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.069538 (+ 285us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.069880 (+ 342us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.069883 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.069885 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.070024 (+ 139us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.070483 (+ 459us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:48.070492 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:48.070762 (+ 270us) client_negotiation.cc:770] Sending connection context
0430 20:01:48.070962 (+ 200us) client_negotiation.cc:241] Negotiation successful
0430 20:01:48.071258 (+ 296us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":173,"threads_started":1}
I20260430 20:01:48.074137 13103 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34973
I20260430 20:01:48.074558 13103 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:48.075516 13103 heartbeater.cc:507] Master 127.10.46.62:34973 requested a full tablet report, sending...
I20260430 20:01:48.078068 12910 ts_manager.cc:194] Registered new tserver with Master: 560d8fc92927495da4839e7392ebba32 (127.10.46.1:33893)
I20260430 20:01:48.079389 12910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:50973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:48.120784 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34973
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:48.250077 13110 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:48.250317 13110 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:48.250379 13110 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:48.253724 13110 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:48.253796 13110 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:48.253883 13110 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:48.258558 13110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:48.259711 13110 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:48.260622 13110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:48.267738 13115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:48.267808 13116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:48.267858 13110 server_base.cc:1061] running on GCE node
W20260430 20:01:48.267738 13118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:48.268465 13110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:48.269054 13110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:48.270277 13110 hybrid_clock.cc:648] HybridClock initialized: now 1777579308270228 us; error 70 us; skew 500 ppm
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579308, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:48.273587 13110 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:48.275127 13110 webserver.cc:492] Webserver started at http://127.10.46.2:46237/ using document root <none> and password file <none>
I20260430 20:01:48.276288 13110 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:48.276427 13110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:48.276894 13110 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:48.279026 13110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "7d25101eaad94531820e66561221e55b"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "a854c3695c9962112f0b2cb30c8d03d3"
server_key_iv: "4ac4add95c1def9af3b7c19b9ce2f18e"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.279569 13110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "7d25101eaad94531820e66561221e55b"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "a854c3695c9962112f0b2cb30c8d03d3"
server_key_iv: "4ac4add95c1def9af3b7c19b9ce2f18e"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.283442 13110 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260430 20:01:48.286279 13125 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.287472 13110 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:48.287626 13110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "7d25101eaad94531820e66561221e55b"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "a854c3695c9962112f0b2cb30c8d03d3"
server_key_iv: "4ac4add95c1def9af3b7c19b9ce2f18e"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.287742 13110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:48.322707 13110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:48.325901 13110 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:48.326131 13110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:48.326730 13110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:48.327667 13110 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:48.327742 13110 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.327834 13110 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:48.327881 13110 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.339458 13110 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:35953
I20260430 20:01:48.339476 13238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:35953 every 8 connection(s)
I20260430 20:01:48.340569 13110 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:48.349009 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13110
I20260430 20:01:48.349117 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:48.349388 10424 external_mini_cluster.cc:1468] Setting key 827ee94376b3483b0521069926a729f9
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579308, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:48.355618 12949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.343088 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:58969 (local address 127.10.46.62:34973)
0430 20:01:48.343279 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:48.343283 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:48.344058 (+ 775us) server_negotiation.cc:408] Connection header received
0430 20:01:48.345065 (+ 1007us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:48.345069 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:48.345131 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:48.345219 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:48.346938 (+ 1719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.347484 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.348306 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.348459 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.350981 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:48.350999 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:48.351003 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:48.351031 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:48.353430 (+ 2399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.353975 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.353979 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.353981 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.354039 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.354414 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.354417 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.354419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.354579 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:48.354673 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:48.355340 (+ 667us) server_negotiation.cc:300] Negotiation successful
0430 20:01:48.355479 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260430 20:01:48.356989 13241 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.343342 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34973 (local address 127.10.46.2:58969)
0430 20:01:48.343897 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:48.343934 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:48.344702 (+ 768us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:48.345419 (+ 717us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:48.345427 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:48.345843 (+ 416us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:48.346739 (+ 896us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.346751 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.347637 (+ 886us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.347640 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:48.348170 (+ 530us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.348179 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.348356 (+ 177us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.348965 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:48.348985 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:48.350804 (+ 1819us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:48.353575 (+ 2771us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.353583 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.353602 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.353863 (+ 261us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.354156 (+ 293us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.354162 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.354167 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.354305 (+ 138us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.354703 (+ 398us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:48.354713 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:48.355018 (+ 305us) client_negotiation.cc:770] Sending connection context
0430 20:01:48.355268 (+ 250us) client_negotiation.cc:241] Negotiation successful
0430 20:01:48.355549 (+ 281us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":102,"threads_started":1}
I20260430 20:01:48.358441 13239 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34973
I20260430 20:01:48.358771 13239 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:48.359596 13239 heartbeater.cc:507] Master 127.10.46.62:34973 requested a full tablet report, sending...
I20260430 20:01:48.360918 12910 ts_manager.cc:194] Registered new tserver with Master: 7d25101eaad94531820e66561221e55b (127.10.46.2:35953)
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:48.361583 12910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:58969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:48.417795 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34973
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:48.537906 13246 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:48.538204 13246 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:48.538304 13246 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:48.542219 13246 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:48.542335 13246 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:48.542447 13246 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:48.548341 13246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:48.549636 13246 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:48.550853 13246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:48.558209 13252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:48.558948 13251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:48.559092 13254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:48.559161 13246 server_base.cc:1061] running on GCE node
I20260430 20:01:48.559978 13246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:48.560547 13246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:48.561725 13246 hybrid_clock.cc:648] HybridClock initialized: now 1777579308561692 us; error 54 us; skew 500 ppm
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579308, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:48.564841 13246 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:48.565986 13246 webserver.cc:492] Webserver started at http://127.10.46.3:44355/ using document root <none> and password file <none>
I20260430 20:01:48.566603 13246 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:48.566658 13246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:48.566886 13246 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:48.568950 13246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "5bced6e97b254e8f80c3af09bff6a9d3"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "f94e6187603556ede2251da9c20371fa"
server_key_iv: "68bdfe45ab0b873d10d99f4d035b97e8"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.570034 13246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "5bced6e97b254e8f80c3af09bff6a9d3"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "f94e6187603556ede2251da9c20371fa"
server_key_iv: "68bdfe45ab0b873d10d99f4d035b97e8"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.574242 13246 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260430 20:01:48.576789 13261 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.579092 13246 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:01:48.579267 13246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "5bced6e97b254e8f80c3af09bff6a9d3"
format_stamp: "Formatted at 2026-04-30 20:01:48 on dist-test-slave-0bdw"
server_key: "f94e6187603556ede2251da9c20371fa"
server_key_iv: "68bdfe45ab0b873d10d99f4d035b97e8"
server_key_version: "encryptionkey@0"
I20260430 20:01:48.579397 13246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:48.604856 13246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:48.608695 13246 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:48.609006 13246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:48.609818 13246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:48.611271 13246 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:48.611414 13246 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.611500 13246 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:48.611543 13246 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:48.622892 13246 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:43897
I20260430 20:01:48.622905 13374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:43897 every 8 connection(s)
I20260430 20:01:48.623951 13246 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:48.626005 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13246
I20260430 20:01:48.626163 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:48.626464 10424 external_mini_cluster.cc:1468] Setting key d3644bad4a1f7cc7c80f3783e8295bd0
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579308, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:48.639694 12949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.626265 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:56681 (local address 127.10.46.62:34973)
0430 20:01:48.626589 (+ 324us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:48.626595 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:48.627613 (+ 1018us) server_negotiation.cc:408] Connection header received
0430 20:01:48.628564 (+ 951us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:48.628568 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:48.628628 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:48.628736 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:48.630354 (+ 1618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.630889 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.631683 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.631921 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.634620 (+ 2699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:48.634648 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:48.634651 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:48.634679 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:48.636416 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.637452 (+ 1036us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.637456 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.637460 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.637515 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:48.637829 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:48.637833 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:48.637835 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:48.638238 (+ 403us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:48.638473 (+ 235us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:48.639325 (+ 852us) server_negotiation.cc:300] Negotiation successful
0430 20:01:48.639501 (+ 176us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260430 20:01:48.640431 13377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:48.626953 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34973 (local address 127.10.46.3:56681)
0430 20:01:48.627477 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:48.627519 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:48.628368 (+ 849us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:48.628904 (+ 536us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:48.628913 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:48.629333 (+ 420us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:48.630199 (+ 866us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.630214 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.631020 (+ 806us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:48.631026 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:48.631524 (+ 498us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:48.631534 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:48.631784 (+ 250us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:48.632571 (+ 787us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:48.632591 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:48.634437 (+ 1846us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:48.636573 (+ 2136us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.636706 (+ 133us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.636730 (+ 24us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.637293 (+ 563us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.637614 (+ 321us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:48.637617 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:48.637619 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:48.637728 (+ 109us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:48.638483 (+ 755us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:48.638495 (+ 12us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:48.638878 (+ 383us) client_negotiation.cc:770] Sending connection context
0430 20:01:48.639263 (+ 385us) client_negotiation.cc:241] Negotiation successful
0430 20:01:48.639537 (+ 274us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":162,"threads_started":1}
I20260430 20:01:48.641736 13375 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34973
I20260430 20:01:48.642011 13375 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:48.642907 13375 heartbeater.cc:507] Master 127.10.46.62:34973 requested a full tablet report, sending...
I20260430 20:01:48.644400 12910 ts_manager.cc:194] Registered new tserver with Master: 5bced6e97b254e8f80c3af09bff6a9d3 (127.10.46.3:43897)
I20260430 20:01:48.645460 12910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:56681
I20260430 20:01:48.654546 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 20:01:48.669034 12949 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:01:48.667073 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.10.46.62:34973)
0430 20:01:48.667268 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:48.667272 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:48.667289 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:01:48.667462 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:48.667466 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:48.667520 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:48.667619 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:48.668866 (+ 1247us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33778: BlockingRecv error: recv got EOF from 127.0.0.1:33778 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260430 20:01:48.670579 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12974
I20260430 20:01:48.677184 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13110
I20260430 20:01:48.683516 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13246
I20260430 20:01:48.692951 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 12878
2026-04-30T20:01:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4211 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[13387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[13387](info): set up 2 sockets
Apr 30 20:01:48 dist-test-slave-0bdw krb5kdc[13387](info): commencing operation
krb5kdc: starting...
W20260430 20:01:50.805100 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.088s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:50 dist-test-slave-0bdw krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:50.980952 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34345
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:51.092928 13403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:51.093246 13403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:51.093348 13403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:51.096909 13403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:51.097031 13403 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:51.097077 13403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:51.097121 13403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:51.097167 13403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:51.102840 13403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:51.104146 13403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:51.105098 13403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:51.111460 13408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.111459 13409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.111451 13411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:51.112164 13403 server_base.cc:1061] running on GCE node
I20260430 20:01:51.113194 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:51.114257 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:51.115442 13403 hybrid_clock.cc:648] HybridClock initialized: now 1777579311115395 us; error 66 us; skew 500 ppm
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579311, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:51.118974 13403 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:51.120275 13403 webserver.cc:492] Webserver started at http://127.10.46.62:36485/ using document root <none> and password file <none>
I20260430 20:01:51.120873 13403 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:51.120925 13403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:51.121150 13403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:51.122960 13403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "b39ef7582b694df0b16c6ac1259cbaed"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "8aef3e4310475e4cb0e1761b85403f6e"
server_key_iv: "c2e0c81b56c639ba0ccb001fc673aff3"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.123509 13403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "b39ef7582b694df0b16c6ac1259cbaed"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "8aef3e4310475e4cb0e1761b85403f6e"
server_key_iv: "c2e0c81b56c639ba0ccb001fc673aff3"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.127378 13403 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:51.130214 13418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.131753 13403 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:01:51.131978 13403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "b39ef7582b694df0b16c6ac1259cbaed"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "8aef3e4310475e4cb0e1761b85403f6e"
server_key_iv: "c2e0c81b56c639ba0ccb001fc673aff3"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.132107 13403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:51.157068 13403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:51.161217 13403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:51.161477 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:51.171066 13403 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:34345
I20260430 20:01:51.171077 13470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:34345 every 8 connection(s)
I20260430 20:01:51.172633 13403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:51.177254 13471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:51.179433 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13403
I20260430 20:01:51.179571 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:51.179915 10424 external_mini_cluster.cc:1468] Setting key a0c514693a6d74669acb5c31af6a1544
I20260430 20:01:51.184903 13471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed: Bootstrap starting.
I20260430 20:01:51.187712 13471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:51.188807 13471 log.cc:826] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed: Log is configured to *not* fsync() on all Append() calls
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:51.191303 13471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed: No bootstrap required, opened a new log
I20260430 20:01:51.194933 13471 raft_consensus.cc:359] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } }
I20260430 20:01:51.195235 13471 raft_consensus.cc:385] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:51.195329 13471 raft_consensus.cc:740] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39ef7582b694df0b16c6ac1259cbaed, State: Initialized, Role: FOLLOWER
I20260430 20:01:51.195905 13471 consensus_queue.cc:260] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } }
I20260430 20:01:51.196084 13471 raft_consensus.cc:399] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:51.196156 13471 raft_consensus.cc:493] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:51.196257 13471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:51.197510 13474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:51.181746 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.10.46.62:34345)
0430 20:01:51.182288 (+ 542us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:51.182299 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:51.182336 (+ 37us) server_negotiation.cc:408] Connection header received
0430 20:01:51.182917 (+ 581us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:51.182933 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:51.183230 (+ 297us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:51.183500 (+ 270us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:51.184839 (+ 1339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.185925 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:51.187027 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.187410 (+ 383us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:51.190803 (+ 3393us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:51.190834 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:51.190850 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:51.190891 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:51.194105 (+ 3214us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.194892 (+ 787us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.194899 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.194906 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.195016 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.195506 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.195510 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.195512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.196013 (+ 501us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:51.196252 (+ 239us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:51.196622 (+ 370us) server_negotiation.cc:300] Negotiation successful
0430 20:01:51.196890 (+ 268us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":157,"threads_started":1}
I20260430 20:01:51.198029 13471 raft_consensus.cc:515] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } }
I20260430 20:01:51.198408 13471 leader_election.cc:304] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b39ef7582b694df0b16c6ac1259cbaed; no voters:
I20260430 20:01:51.198760 13471 leader_election.cc:290] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:51.198930 13476 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:51.199142 13476 raft_consensus.cc:697] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [term 1 LEADER]: Becoming Leader. State: Replica: b39ef7582b694df0b16c6ac1259cbaed, State: Running, Role: LEADER
I20260430 20:01:51.199503 13476 consensus_queue.cc:237] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } }
I20260430 20:01:51.200070 13471 sys_catalog.cc:565] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:51.201473 13478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39ef7582b694df0b16c6ac1259cbaed. Latest consensus state: current_term: 1 leader_uuid: "b39ef7582b694df0b16c6ac1259cbaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } } }
I20260430 20:01:51.201609 13478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:51.201665 13477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b39ef7582b694df0b16c6ac1259cbaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39ef7582b694df0b16c6ac1259cbaed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34345 } } }
I20260430 20:01:51.201777 13477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:51.202216 13483 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:51.205436 13483 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:51.213059 13483 catalog_manager.cc:1357] Generated new cluster ID: 006ef5467cc342f792c8f3995c404614
I20260430 20:01:51.213199 13483 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:51.231618 13483 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:51.233098 13483 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:51.240520 13483 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b39ef7582b694df0b16c6ac1259cbaed: Generated new TSK 0
I20260430 20:01:51.241369 13483 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:51.318693 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34345
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:51.437625 13499 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:51.437911 13499 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:51.437975 13499 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:51.442047 13499 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:51.442139 13499 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:51.442224 13499 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:51.447490 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:51.448733 13499 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:51.449558 13499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:51.459034 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.459136 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.459067 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:51.459429 13499 server_base.cc:1061] running on GCE node
I20260430 20:01:51.460005 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:51.460680 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:51.461951 13499 hybrid_clock.cc:648] HybridClock initialized: now 1777579311461897 us; error 66 us; skew 500 ppm
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579311, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:51.465422 13499 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:51.466778 13499 webserver.cc:492] Webserver started at http://127.10.46.1:39639/ using document root <none> and password file <none>
I20260430 20:01:51.467409 13499 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:51.467490 13499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:51.467702 13499 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:51.469570 13499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "1a79e53a59444b938463818dfe19b642"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "9837c1c3c207c2b78577697d63297066"
server_key_iv: "98c270a58ca80dfb50213affacd99d3d"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.470466 13499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "1a79e53a59444b938463818dfe19b642"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "9837c1c3c207c2b78577697d63297066"
server_key_iv: "98c270a58ca80dfb50213affacd99d3d"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.474337 13499 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:01:51.477219 13514 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.478405 13499 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:51.478574 13499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "1a79e53a59444b938463818dfe19b642"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "9837c1c3c207c2b78577697d63297066"
server_key_iv: "98c270a58ca80dfb50213affacd99d3d"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.478699 13499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:51.488586 13499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:51.491919 13499 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:51.492142 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:51.492759 13499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:51.493710 13499 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:51.493784 13499 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.493883 13499 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:51.493929 13499 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.506758 13499 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:39273
I20260430 20:01:51.506810 13627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:39273 every 8 connection(s)
I20260430 20:01:51.507889 13499 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:51.516188 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13499
I20260430 20:01:51.516357 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:51.516603 10424 external_mini_cluster.cc:1468] Setting key b21debe9e82de89daf5d435749035a4c
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579311, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:51.523399 13474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:51.510722 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:42499 (local address 127.10.46.62:34345)
0430 20:01:51.510959 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:51.510963 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:51.511605 (+ 642us) server_negotiation.cc:408] Connection header received
0430 20:01:51.512600 (+ 995us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:51.512604 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:51.512669 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:51.512773 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:51.514418 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.515097 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:51.515949 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.516146 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:51.519716 (+ 3570us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:51.519744 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:51.519746 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:51.519821 (+ 75us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:51.521440 (+ 1619us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.521962 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.521967 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.521969 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.522022 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.522320 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.522324 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.522326 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.522492 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:51.522598 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:51.523097 (+ 499us) server_negotiation.cc:300] Negotiation successful
0430 20:01:51.523229 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260430 20:01:51.524123 13630 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:51.510996 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34345 (local address 127.10.46.1:42499)
0430 20:01:51.511450 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:51.511484 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:51.512378 (+ 894us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:51.512944 (+ 566us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:51.512953 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:51.513353 (+ 400us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:51.513848 (+ 495us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:51.513860 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.515294 (+ 1434us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:51.515299 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:51.515734 (+ 435us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:51.515742 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.516057 (+ 315us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:51.517334 (+ 1277us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:51.517354 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:51.519536 (+ 2182us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:51.521581 (+ 2045us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:51.521588 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:51.521602 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:51.521856 (+ 254us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:51.522131 (+ 275us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:51.522133 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:51.522135 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:51.522226 (+ 91us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:51.522609 (+ 383us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:51.522617 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:51.522846 (+ 229us) client_negotiation.cc:770] Sending connection context
0430 20:01:51.523052 (+ 206us) client_negotiation.cc:241] Negotiation successful
0430 20:01:51.523308 (+ 256us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260430 20:01:51.525472 13628 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34345
I20260430 20:01:51.525755 13628 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:01:51.526383 13628 heartbeater.cc:507] Master 127.10.46.62:34345 requested a full tablet report, sending...
I20260430 20:01:51.528051 13433 ts_manager.cc:194] Registered new tserver with Master: 1a79e53a59444b938463818dfe19b642 (127.10.46.1:39273)
I20260430 20:01:51.529412 13433 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:42499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:51.585424 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34345
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:51.726576 13635 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:51.727048 13635 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:51.727129 13635 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:51.730908 13635 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:51.730993 13635 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:51.731078 13635 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:51.736176 13635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:51.737351 13635 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:51.738265 13635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:51.746309 13640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.746734 13641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:51.746323 13643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:51.746778 13635 server_base.cc:1061] running on GCE node
I20260430 20:01:51.747273 13635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:51.748382 13635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:51.749754 13635 hybrid_clock.cc:648] HybridClock initialized: now 1777579311749777 us; error 88 us; skew 500 ppm
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579311, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:51.753422 13635 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:51.754729 13635 webserver.cc:492] Webserver started at http://127.10.46.2:42529/ using document root <none> and password file <none>
I20260430 20:01:51.755357 13635 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:51.755437 13635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:51.755652 13635 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:51.757856 13635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "b368fa8fd6af4b15b68739ebcfb5bc87"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "70664bb1f92de7f7bbe24a39598b2f8e"
server_key_iv: "2e70b137dea37953f5d370f553ee0158"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.758394 13635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "b368fa8fd6af4b15b68739ebcfb5bc87"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "70664bb1f92de7f7bbe24a39598b2f8e"
server_key_iv: "2e70b137dea37953f5d370f553ee0158"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.762651 13635 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260430 20:01:51.766157 13650 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.768185 13635 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:01:51.768373 13635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "b368fa8fd6af4b15b68739ebcfb5bc87"
format_stamp: "Formatted at 2026-04-30 20:01:51 on dist-test-slave-0bdw"
server_key: "70664bb1f92de7f7bbe24a39598b2f8e"
server_key_iv: "2e70b137dea37953f5d370f553ee0158"
server_key_version: "encryptionkey@0"
I20260430 20:01:51.768520 13635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:51.787753 13635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:51.791335 13635 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:51.791585 13635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:51.792308 13635 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:51.793270 13635 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:51.793347 13635 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.793417 13635 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:51.793460 13635 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:51.804352 13635 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:42765
I20260430 20:01:51.804553 13763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:42765 every 8 connection(s)
I20260430 20:01:51.805471 13635 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:51.814566 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13635
I20260430 20:01:51.814726 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:51.815035 10424 external_mini_cluster.cc:1468] Setting key 5a4c619bd307cddd91c8601373a105a4
Apr 30 20:01:51 dist-test-slave-0bdw krb5kdc[13387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579311, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:51.821817 13474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:51.808276 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:46735 (local address 127.10.46.62:34345)
0430 20:01:51.808476 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:51.808481 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:51.809508 (+ 1027us) server_negotiation.cc:408] Connection header received
0430 20:01:51.810395 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:51.810398 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:51.810452 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:51.810538 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:51.812133 (+ 1595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.812748 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:51.813640 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.814126 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:51.817918 (+ 3792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:51.817938 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:51.817940 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:51.817970 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:51.819667 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.820319 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.820322 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.820324 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.820370 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:51.820699 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:51.820702 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:51.820703 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:51.820870 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:51.820981 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:51.821504 (+ 523us) server_negotiation.cc:300] Negotiation successful
0430 20:01:51.821641 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260430 20:01:51.822496 13766 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:51.808688 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34345 (local address 127.10.46.2:46735)
0430 20:01:51.809223 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:51.809260 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:51.810188 (+ 928us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:51.810733 (+ 545us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:51.810742 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:51.811201 (+ 459us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:51.811942 (+ 741us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:51.811957 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.812927 (+ 970us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:51.812930 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:51.813345 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:51.813352 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:51.813529 (+ 177us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:51.815096 (+ 1567us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:51.815125 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:51.817716 (+ 2591us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:51.819843 (+ 2127us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:51.819851 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:51.819865 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:51.820184 (+ 319us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:51.820479 (+ 295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:51.820482 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:51.820484 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:51.820577 (+ 93us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:51.820979 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:51.820986 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:51.821238 (+ 252us) client_negotiation.cc:770] Sending connection context
0430 20:01:51.821453 (+ 215us) client_negotiation.cc:241] Negotiation successful
0430 20:01:51.821702 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":149,"threads_started":1}
I20260430 20:01:51.823917 13764 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34345
I20260430 20:01:51.824198 13764 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:51.824731 13764 heartbeater.cc:507] Master 127.10.46.62:34345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:01:51.826211 13433 ts_manager.cc:194] Registered new tserver with Master: b368fa8fd6af4b15b68739ebcfb5bc87 (127.10.46.2:42765)
I20260430 20:01:51.826849 13433 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:46735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:51.881220 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34345
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:52.008299 13771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:52.008565 13771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:52.008625 13771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:52.012295 13771 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:52.012377 13771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:52.012506 13771 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:52.017452 13771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:52.018618 13771 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:52.019506 13771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:52.026191 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:52.026147 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:52.026147 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:52.026556 13771 server_base.cc:1061] running on GCE node
I20260430 20:01:52.027168 13771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:52.028501 13771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:52.029721 13771 hybrid_clock.cc:648] HybridClock initialized: now 1777579312029661 us; error 82 us; skew 500 ppm
Apr 30 20:01:52 dist-test-slave-0bdw krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579312, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:52.033157 13771 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:52.034430 13771 webserver.cc:492] Webserver started at http://127.10.46.3:36671/ using document root <none> and password file <none>
I20260430 20:01:52.035080 13771 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:52.035135 13771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:52.035346 13771 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:52.038095 13771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "1c5b29e557c64edf8023c9e09df03115"
format_stamp: "Formatted at 2026-04-30 20:01:52 on dist-test-slave-0bdw"
server_key: "d41a8162f8648fd6b43acaffaec266fe"
server_key_iv: "a5a253f5ff9f3e8bf7a7044ed9e0b0d3"
server_key_version: "encryptionkey@0"
I20260430 20:01:52.038699 13771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "1c5b29e557c64edf8023c9e09df03115"
format_stamp: "Formatted at 2026-04-30 20:01:52 on dist-test-slave-0bdw"
server_key: "d41a8162f8648fd6b43acaffaec266fe"
server_key_iv: "a5a253f5ff9f3e8bf7a7044ed9e0b0d3"
server_key_version: "encryptionkey@0"
I20260430 20:01:52.042460 13771 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260430 20:01:52.045030 13786 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:52.046226 13771 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 20:01:52.046399 13771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "1c5b29e557c64edf8023c9e09df03115"
format_stamp: "Formatted at 2026-04-30 20:01:52 on dist-test-slave-0bdw"
server_key: "d41a8162f8648fd6b43acaffaec266fe"
server_key_iv: "a5a253f5ff9f3e8bf7a7044ed9e0b0d3"
server_key_version: "encryptionkey@0"
I20260430 20:01:52.046526 13771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:52.066398 13771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:52.070212 13771 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:52.070444 13771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:52.071066 13771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:52.072244 13771 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:52.072324 13771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:52.072391 13771 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:52.072432 13771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:52.083586 13771 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:45455
I20260430 20:01:52.083634 13899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:45455 every 8 connection(s)
I20260430 20:01:52.084560 13771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:52.088984 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13771
I20260430 20:01:52.089099 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:52.089397 10424 external_mini_cluster.cc:1468] Setting key fe30ab48d24ea5fc9e10e0d584e84cd4
Apr 30 20:01:52 dist-test-slave-0bdw krb5kdc[13387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579312, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:52.098222 13474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:52.086571 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:40427 (local address 127.10.46.62:34345)
0430 20:01:52.086748 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:52.086751 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:52.087388 (+ 637us) server_negotiation.cc:408] Connection header received
0430 20:01:52.088291 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:52.088294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:52.088357 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:52.088461 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:52.089835 (+ 1374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:52.090490 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:52.091476 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:52.091766 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:52.094634 (+ 2868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:52.094656 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:52.094658 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:52.094691 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:52.096250 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:52.096804 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:52.096808 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:52.096810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:52.096860 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:52.097172 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:52.097175 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:52.097177 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:52.097349 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:52.097440 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:52.097958 (+ 518us) server_negotiation.cc:300] Negotiation successful
0430 20:01:52.098093 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260430 20:01:52.098841 13902 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:52.086822 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34345 (local address 127.10.46.3:40427)
0430 20:01:52.087234 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:52.087277 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:52.088086 (+ 809us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:52.088628 (+ 542us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:52.088637 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:52.089041 (+ 404us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:52.089641 (+ 600us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:52.089653 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:52.090627 (+ 974us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:52.090631 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:52.091209 (+ 578us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:52.091218 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:52.091438 (+ 220us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:52.092821 (+ 1383us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:52.092841 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:52.094312 (+ 1471us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:52.096400 (+ 2088us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:52.096405 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:52.096415 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:52.096685 (+ 270us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:52.096950 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:52.096952 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:52.096954 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:52.097041 (+ 87us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:52.097444 (+ 403us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:52.097448 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:52.097677 (+ 229us) client_negotiation.cc:770] Sending connection context
0430 20:01:52.097862 (+ 185us) client_negotiation.cc:241] Negotiation successful
0430 20:01:52.098094 (+ 232us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
I20260430 20:01:52.100311 13900 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34345
I20260430 20:01:52.100546 13900 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:52.101107 13900 heartbeater.cc:507] Master 127.10.46.62:34345 requested a full tablet report, sending...
I20260430 20:01:52.102401 13433 ts_manager.cc:194] Registered new tserver with Master: 1c5b29e557c64edf8023c9e09df03115 (127.10.46.3:45455)
I20260430 20:01:52.103045 13433 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:40427
I20260430 20:01:52.103878 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 20:01:52.239671 13474 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:01:52.235624 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.10.46.62:34345)
0430 20:01:52.235897 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:52.235902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:52.236963 (+ 1061us) server_negotiation.cc:408] Connection header received
0430 20:01:52.238109 (+ 1146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:52.238112 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:52.238168 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:52.238250 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:52.239527 (+ 1277us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37644: BlockingRecv error: recv got EOF from 127.0.0.1:37644 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260430 20:01:52.241457 13474 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:01:52.240445 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.10.46.62:34345)
0430 20:01:52.240576 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:52.240582 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:52.240598 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:01:52.240761 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:52.240765 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:52.240817 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:52.240956 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:52.241361 (+ 405us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37650: BlockingRecv error: recv got EOF from 127.0.0.1:37650 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260430 20:01:52.243145 13474 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:01:52.241992 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.10.46.62:34345)
0430 20:01:52.242158 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:52.242161 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:52.242174 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:01:52.242205 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:52.242208 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:52.242243 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:52.242321 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:52.243016 (+ 695us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37654: BlockingRecv error: recv got EOF from 127.0.0.1:37654 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260430 20:01:52.249722 13474 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:01:52.248662 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.10.46.62:34345)
0430 20:01:52.248859 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:52.248863 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:52.248881 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:01:52.248953 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:52.248956 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:52.249004 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:52.249097 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:52.249618 (+ 521us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37670: BlockingRecv error: recv got EOF from 127.0.0.1:37670 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260430 20:01:52.260166 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13499
I20260430 20:01:52.266584 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13635
I20260430 20:01:52.273016 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13771
I20260430 20:01:52.279008 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13403
2026-04-30T20:01:52Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3586 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:52 dist-test-slave-0bdw krb5kdc[13923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:52 dist-test-slave-0bdw krb5kdc[13923](info): set up 2 sockets
Apr 30 20:01:52 dist-test-slave-0bdw krb5kdc[13923](info): commencing operation
krb5kdc: starting...
W20260430 20:01:54.444706 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:54 dist-test-slave-0bdw krb5kdc[13923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:54.666957 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37177
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:54.815404 13939 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:54.815980 13939 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:54.816138 13939 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:54.820756 13939 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:54.820869 13939 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:54.820895 13939 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:54.820916 13939 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:54.820940 13939 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:54.826035 13939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:54.827658 13939 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:54.828832 13939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:54.837122 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:54.837122 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:54.837329 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:54.838342 13939 server_base.cc:1061] running on GCE node
I20260430 20:01:54.839490 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:54.841137 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:54.842440 13939 hybrid_clock.cc:648] HybridClock initialized: now 1777579314842393 us; error 71 us; skew 500 ppm
Apr 30 20:01:54 dist-test-slave-0bdw krb5kdc[13923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:54.845832 13939 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:54.847345 13939 webserver.cc:492] Webserver started at http://127.10.46.62:44815/ using document root <none> and password file <none>
I20260430 20:01:54.848106 13939 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:54.848201 13939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:54.848438 13939 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:54.850754 13939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "2fa399ff8d0f4884a29c5d67d79e8b81"
format_stamp: "Formatted at 2026-04-30 20:01:54 on dist-test-slave-0bdw"
server_key: "d10217a1966e9dd30539e4e8a0889b39"
server_key_iv: "ed1b35a7a72ebb3a64ef50e869a7d9e0"
server_key_version: "encryptionkey@0"
I20260430 20:01:54.851406 13939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "2fa399ff8d0f4884a29c5d67d79e8b81"
format_stamp: "Formatted at 2026-04-30 20:01:54 on dist-test-slave-0bdw"
server_key: "d10217a1966e9dd30539e4e8a0889b39"
server_key_iv: "ed1b35a7a72ebb3a64ef50e869a7d9e0"
server_key_version: "encryptionkey@0"
I20260430 20:01:54.856594 13939 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260430 20:01:54.860013 13954 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:54.861370 13939 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:01:54.861572 13939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "2fa399ff8d0f4884a29c5d67d79e8b81"
format_stamp: "Formatted at 2026-04-30 20:01:54 on dist-test-slave-0bdw"
server_key: "d10217a1966e9dd30539e4e8a0889b39"
server_key_iv: "ed1b35a7a72ebb3a64ef50e869a7d9e0"
server_key_version: "encryptionkey@0"
I20260430 20:01:54.861722 13939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:54.884524 13939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:54.893515 13939 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:54.893841 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:54.903409 14006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:37177 every 8 connection(s)
I20260430 20:01:54.903416 13939 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:37177
I20260430 20:01:54.904731 13939 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:54.907235 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 13939
I20260430 20:01:54.907374 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:54.907618 10424 external_mini_cluster.cc:1468] Setting key fb283d8bbc44b7f92f13cec28aa2b113
I20260430 20:01:54.908109 14007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:54.916392 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: Bootstrap starting.
Apr 30 20:01:54 dist-test-slave-0bdw krb5kdc[13923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:54.920751 14007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:54.921694 14007 log.cc:826] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:54.923684 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: No bootstrap required, opened a new log
I20260430 20:01:54.925074 14010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:54.908943 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.10.46.62:37177)
0430 20:01:54.909408 (+ 465us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:54.909419 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:54.909455 (+ 36us) server_negotiation.cc:408] Connection header received
0430 20:01:54.910601 (+ 1146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:54.910620 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:54.910924 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:54.911216 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:54.912888 (+ 1672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:54.913869 (+ 981us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:54.915270 (+ 1401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:54.915575 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:54.919929 (+ 4354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:54.919966 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:54.919981 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:54.920029 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:54.922248 (+ 2219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:54.922773 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:54.922778 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:54.922783 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:54.922857 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:54.923148 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:54.923152 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:54.923154 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:54.923498 (+ 344us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:54.923700 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:54.924067 (+ 367us) server_negotiation.cc:300] Negotiation successful
0430 20:01:54.924338 (+ 271us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":175,"threads_started":1}
I20260430 20:01:54.926957 14007 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } }
I20260430 20:01:54.927197 14007 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:54.927248 14007 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa399ff8d0f4884a29c5d67d79e8b81, State: Initialized, Role: FOLLOWER
I20260430 20:01:54.927655 14007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } }
I20260430 20:01:54.927841 14007 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:54.927932 14007 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:54.928028 14007 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:54.928985 14007 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } }
I20260430 20:01:54.929164 10424 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 20:01:54.929412 14007 leader_election.cc:304] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa399ff8d0f4884a29c5d67d79e8b81; no voters:
I20260430 20:01:54.930050 14007 leader_election.cc:290] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:54.930452 14012 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:54.930996 14012 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa399ff8d0f4884a29c5d67d79e8b81, State: Running, Role: LEADER
I20260430 20:01:54.931631 14012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } }
I20260430 20:01:54.932049 14007 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:54.934533 14013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } } }
I20260430 20:01:54.934698 14013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:54.935011 14014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa399ff8d0f4884a29c5d67d79e8b81. Latest consensus state: current_term: 1 leader_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa399ff8d0f4884a29c5d67d79e8b81" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37177 } } }
I20260430 20:01:54.935175 14014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81 [sys.catalog]: This master's current role is: LEADER
W20260430 20:01:54.939715 14028 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:01:54.939885 14028 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:01:54.939955 14022 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:54.940891 14022 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:54.947283 14022 catalog_manager.cc:1357] Generated new cluster ID: cb75eb8237c04d05949e90d3d8284312
I20260430 20:01:54.947481 14022 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:54.950944 14010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:54.944547 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.10.46.62:37177)
0430 20:01:54.944682 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:54.944686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:54.944936 (+ 250us) server_negotiation.cc:408] Connection header received
0430 20:01:54.945067 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:54.945071 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:54.945130 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:54.945237 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:54.946070 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:54.946833 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:54.947809 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:54.948021 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:54.948731 (+ 710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:54.948746 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:54.948750 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:01:54.950234 (+ 1484us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:54.950279 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:01:54.950300 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:54.950411 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:54.950712 (+ 301us) server_negotiation.cc:300] Negotiation successful
0430 20:01:54.950774 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 20:01:54.960657 14022 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:54.961898 14022 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:54.977571 14022 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2fa399ff8d0f4884a29c5d67d79e8b81: Generated new TSK 0
I20260430 20:01:54.978686 14022 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:01:54.990806 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 13939
2026-04-30T20:01:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2713 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:55 dist-test-slave-0bdw krb5kdc[14040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:55 dist-test-slave-0bdw krb5kdc[14040](info): set up 2 sockets
Apr 30 20:01:55 dist-test-slave-0bdw krb5kdc[14040](info): commencing operation
krb5kdc: starting...
W20260430 20:01:57.165210 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.148s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:01:57 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:01:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:57.349720 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:36789
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:36789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:57.484179 14056 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:57.484642 14056 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:57.485090 14056 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:57.489436 14056 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:01:57.489737 14056 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:57.489797 14056 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:57.489862 14056 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:01:57.489924 14056 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:01:57.495142 14056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:36789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:36789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:57.496500 14056 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:57.497510 14056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:57.504876 14061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:57.504928 14062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:57.504890 14064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:57.505540 14056 server_base.cc:1061] running on GCE node
I20260430 20:01:57.506177 14056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:57.507323 14056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:57.510900 14056 hybrid_clock.cc:648] HybridClock initialized: now 1777579317510750 us; error 56 us; skew 500 ppm
Apr 30 20:01:57 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:57.516402 14056 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:01:57.518113 14056 webserver.cc:492] Webserver started at http://127.10.46.62:40633/ using document root <none> and password file <none>
I20260430 20:01:57.518913 14056 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:57.519013 14056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:57.519220 14056 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:57.521180 14056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "e4f48237dd2f4f77823bfb3c59c57dd7"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "72edcde458432675270abc5dc94a2897"
server_key_iv: "3f64ed4b823c7ae42a7deefd456f262a"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.521673 14056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "e4f48237dd2f4f77823bfb3c59c57dd7"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "72edcde458432675270abc5dc94a2897"
server_key_iv: "3f64ed4b823c7ae42a7deefd456f262a"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.526508 14056 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260430 20:01:57.530337 14071 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:57.531991 14056 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:57.532380 14056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "e4f48237dd2f4f77823bfb3c59c57dd7"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "72edcde458432675270abc5dc94a2897"
server_key_iv: "3f64ed4b823c7ae42a7deefd456f262a"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.532591 14056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:57.562259 14056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:57.566058 14056 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:57.566314 14056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:57.576155 14123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:36789 every 8 connection(s)
I20260430 20:01:57.576176 14056 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:36789
I20260430 20:01:57.577363 14056 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:01:57.579753 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14056
I20260430 20:01:57.579964 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:01:57.580267 10424 external_mini_cluster.cc:1468] Setting key 58c7e7ce72690c5f0d209677e36002bd
I20260430 20:01:57.582057 14124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:01:57.589066 14124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7: Bootstrap starting.
Apr 30 20:01:57 dist-test-slave-0bdw krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:57.593382 14124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7: Neither blocks nor log segments found. Creating new log.
I20260430 20:01:57.595419 14124 log.cc:826] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7: Log is configured to *not* fsync() on all Append() calls
I20260430 20:01:57.599442 14124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7: No bootstrap required, opened a new log
I20260430 20:01:57.599754 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:57.581724 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.10.46.62:36789)
0430 20:01:57.582548 (+ 824us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:57.582563 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:57.582623 (+ 60us) server_negotiation.cc:408] Connection header received
0430 20:01:57.583561 (+ 938us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:57.583583 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:57.584267 (+ 684us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:57.584744 (+ 477us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:57.585863 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.586651 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:57.587680 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.588066 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:57.591318 (+ 3252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:57.591376 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:57.591391 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:57.591442 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:57.594373 (+ 2931us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:57.595724 (+ 1351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:57.595734 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:57.595743 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:57.596289 (+ 546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:57.596827 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:57.596832 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:57.596834 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:57.597514 (+ 680us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:57.597747 (+ 233us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:57.598554 (+ 807us) server_negotiation.cc:300] Negotiation successful
0430 20:01:57.598879 (+ 325us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":205,"threads_started":1}
I20260430 20:01:57.603935 14124 raft_consensus.cc:359] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } }
I20260430 20:01:57.604291 14124 raft_consensus.cc:385] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:01:57.604377 14124 raft_consensus.cc:740] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f48237dd2f4f77823bfb3c59c57dd7, State: Initialized, Role: FOLLOWER
I20260430 20:01:57.604923 14124 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } }
I20260430 20:01:57.605113 14124 raft_consensus.cc:399] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:01:57.605186 14124 raft_consensus.cc:493] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:01:57.605290 14124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:01:57.606519 14124 raft_consensus.cc:515] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } }
I20260430 20:01:57.607016 14124 leader_election.cc:304] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4f48237dd2f4f77823bfb3c59c57dd7; no voters:
I20260430 20:01:57.607380 14124 leader_election.cc:290] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:01:57.607610 14129 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:01:57.607883 14129 raft_consensus.cc:697] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [term 1 LEADER]: Becoming Leader. State: Replica: e4f48237dd2f4f77823bfb3c59c57dd7, State: Running, Role: LEADER
I20260430 20:01:57.608218 14129 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } }
I20260430 20:01:57.608724 14124 sys_catalog.cc:565] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:01:57.609786 14130 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } } }
I20260430 20:01:57.609923 14130 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:57.610384 14134 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:01:57.610426 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f48237dd2f4f77823bfb3c59c57dd7. Latest consensus state: current_term: 1 leader_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f48237dd2f4f77823bfb3c59c57dd7" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 36789 } } }
I20260430 20:01:57.610534 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7 [sys.catalog]: This master's current role is: LEADER
I20260430 20:01:57.613332 14134 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:01:57.619246 14134 catalog_manager.cc:1357] Generated new cluster ID: 9122c299915240069c6af0c45cba2531
I20260430 20:01:57.619351 14134 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:01:57.627106 14134 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:01:57.628269 14134 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:01:57.637996 14134 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e4f48237dd2f4f77823bfb3c59c57dd7: Generated new TSK 0
I20260430 20:01:57.639252 14134 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:57.713719 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:36789
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:57.847196 14152 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:57.847467 14152 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:57.847528 14152 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:57.851279 14152 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:57.851374 14152 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:57.851462 14152 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:01:57.856488 14152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:57.857718 14152 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:57.858630 14152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:57.866153 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:57.866400 14152 server_base.cc:1061] running on GCE node
W20260430 20:01:57.866153 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:57.866158 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:57.867139 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:57.867880 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:57.869081 14152 hybrid_clock.cc:648] HybridClock initialized: now 1777579317869026 us; error 70 us; skew 500 ppm
Apr 30 20:01:57 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:57.872681 14152 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:01:57.874797 14152 webserver.cc:492] Webserver started at http://127.10.46.1:36561/ using document root <none> and password file <none>
I20260430 20:01:57.875716 14152 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:57.875901 14152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:57.876374 14152 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:57.879489 14152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "829272a0bca24e008db66a114974fca6"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "ca0b41750a913845987c46d384d860ab"
server_key_iv: "191d86f42d07a58d22a8ff07cc79d801"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.880620 14152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "829272a0bca24e008db66a114974fca6"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "ca0b41750a913845987c46d384d860ab"
server_key_iv: "191d86f42d07a58d22a8ff07cc79d801"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.885385 14152 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260430 20:01:57.889133 14167 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:57.890880 14152 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:01:57.891129 14152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "829272a0bca24e008db66a114974fca6"
format_stamp: "Formatted at 2026-04-30 20:01:57 on dist-test-slave-0bdw"
server_key: "ca0b41750a913845987c46d384d860ab"
server_key_iv: "191d86f42d07a58d22a8ff07cc79d801"
server_key_version: "encryptionkey@0"
I20260430 20:01:57.891338 14152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:57.910707 14152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:57.916322 14152 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:57.916549 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:57.917227 14152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:57.918524 14152 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:57.918632 14152 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:57.918689 14152 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:57.918705 14152 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:57.929471 14152 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:37203
I20260430 20:01:57.929487 14280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:37203 every 8 connection(s)
I20260430 20:01:57.930637 14152 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:01:57.934394 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14152
I20260430 20:01:57.934728 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:01:57.935119 10424 external_mini_cluster.cc:1468] Setting key e0216b5f20bb126fb2566cf9aef24a81
Apr 30 20:01:57 dist-test-slave-0bdw krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:01:57.950254 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:57.934098 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:59675 (local address 127.10.46.62:36789)
0430 20:01:57.934612 (+ 514us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:57.934620 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:57.935018 (+ 398us) server_negotiation.cc:408] Connection header received
0430 20:01:57.936458 (+ 1440us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:57.936461 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:57.936512 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:57.936628 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:57.939310 (+ 2682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.940202 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:57.941329 (+ 1127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.941649 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:57.944550 (+ 2901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:57.944581 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:57.944584 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:57.944627 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:57.946642 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:57.947761 (+ 1119us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:57.947892 (+ 131us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:57.947895 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:57.948028 (+ 133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:57.948566 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:57.948570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:57.948572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:57.948764 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:57.948875 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:57.949902 (+ 1027us) server_negotiation.cc:300] Negotiation successful
0430 20:01:57.950060 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:01:57.952306 14283 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:57.934167 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:36789 (local address 127.10.46.1:59675)
0430 20:01:57.934822 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:57.934868 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:57.936124 (+ 1256us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:57.937109 (+ 985us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:57.937121 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:57.938043 (+ 922us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:57.939038 (+ 995us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:57.939059 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.940609 (+ 1550us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:57.940617 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:57.941089 (+ 472us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:57.941099 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:57.941326 (+ 227us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:57.942532 (+ 1206us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:57.942554 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:57.944300 (+ 1746us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:57.946884 (+ 2584us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:57.946892 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:57.946904 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:57.947221 (+ 317us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:57.948250 (+ 1029us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:57.948254 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:57.948256 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:57.948389 (+ 133us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:57.948923 (+ 534us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:57.948929 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:57.949305 (+ 376us) client_negotiation.cc:770] Sending connection context
0430 20:01:57.949829 (+ 524us) client_negotiation.cc:241] Negotiation successful
0430 20:01:57.950285 (+ 456us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":172,"threads_started":1}
I20260430 20:01:57.954803 14281 heartbeater.cc:344] Connected to a master server at 127.10.46.62:36789
I20260430 20:01:57.955310 14281 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:57.956305 14281 heartbeater.cc:507] Master 127.10.46.62:36789 requested a full tablet report, sending...
I20260430 20:01:57.958755 14087 ts_manager.cc:194] Registered new tserver with Master: 829272a0bca24e008db66a114974fca6 (127.10.46.1:37203)
I20260430 20:01:57.960840 14087 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:59675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:58.014143 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:36789
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:58.151124 14288 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:58.151443 14288 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:58.151538 14288 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:58.155356 14288 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:58.155774 14288 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:58.156410 14288 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:01:58.163597 14288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:58.165025 14288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:58.166086 14288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:58.177395 14296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:58.177368 14294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:58.177376 14293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:58.177484 14288 server_base.cc:1061] running on GCE node
I20260430 20:01:58.178162 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:58.179178 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:58.180546 14288 hybrid_clock.cc:648] HybridClock initialized: now 1777579318180441 us; error 122 us; skew 500 ppm
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:58.184902 14288 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:01:58.186766 14288 webserver.cc:492] Webserver started at http://127.10.46.2:44495/ using document root <none> and password file <none>
I20260430 20:01:58.187502 14288 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:58.187619 14288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:58.187937 14288 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:58.190110 14288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "03851b6ad2d947df83c3fee9abe771da"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "636614a3fb92dc6b23d20943e113e42b"
server_key_iv: "8fbbe0470f84cab2df780b71c64404f9"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.190742 14288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "03851b6ad2d947df83c3fee9abe771da"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "636614a3fb92dc6b23d20943e113e42b"
server_key_iv: "8fbbe0470f84cab2df780b71c64404f9"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.196944 14288 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260430 20:01:58.201444 14303 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.202991 14288 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:01:58.203210 14288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "03851b6ad2d947df83c3fee9abe771da"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "636614a3fb92dc6b23d20943e113e42b"
server_key_iv: "8fbbe0470f84cab2df780b71c64404f9"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.203361 14288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:58.226910 14288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:58.231354 14288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:58.231640 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:58.232363 14288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:58.233616 14288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:58.233718 14288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.233790 14288 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:58.233851 14288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.248975 14288 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:44939
I20260430 20:01:58.248993 14416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:44939 every 8 connection(s)
I20260430 20:01:58.250401 14288 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:01:58.253737 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14288
I20260430 20:01:58.254017 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:01:58.254400 10424 external_mini_cluster.cc:1468] Setting key 494c3e89d1b8f64109f82369cb39ce01
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
I20260430 20:01:58.270205 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:58.252759 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:45001 (local address 127.10.46.62:36789)
0430 20:01:58.252963 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:58.252968 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:58.254148 (+ 1180us) server_negotiation.cc:408] Connection header received
0430 20:01:58.255857 (+ 1709us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:58.255861 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:58.255920 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:58.256023 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:58.258855 (+ 2832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.259726 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:58.260772 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.261144 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:58.264543 (+ 3399us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:58.264576 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:58.264578 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:58.264612 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:58.266727 (+ 2115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.267589 (+ 862us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.267593 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.267594 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.267647 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.268395 (+ 748us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.268398 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.268400 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.268570 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:58.268663 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:58.269896 (+ 1233us) server_negotiation.cc:300] Negotiation successful
0430 20:01:58.270059 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260430 20:01:58.271303 14419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:58.252991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:36789 (local address 127.10.46.2:45001)
0430 20:01:58.253791 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:58.253879 (+ 88us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:58.255498 (+ 1619us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:58.256521 (+ 1023us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:58.256712 (+ 191us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:58.257645 (+ 933us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:58.258564 (+ 919us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:58.258582 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.259944 (+ 1362us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:58.259949 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:58.260593 (+ 644us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:58.260607 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.260886 (+ 279us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:58.262547 (+ 1661us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:58.262571 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:58.264312 (+ 1741us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:58.267096 (+ 2784us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:58.267107 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:58.267125 (+ 18us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:58.267420 (+ 295us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:58.268001 (+ 581us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:58.268004 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:58.268006 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:58.268116 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:58.268755 (+ 639us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:58.268765 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:58.269375 (+ 610us) client_negotiation.cc:770] Sending connection context
0430 20:01:58.269804 (+ 429us) client_negotiation.cc:241] Negotiation successful
0430 20:01:58.270189 (+ 385us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":112,"threads_started":1}
I20260430 20:01:58.273253 14417 heartbeater.cc:344] Connected to a master server at 127.10.46.62:36789
I20260430 20:01:58.273837 14417 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:58.274490 14417 heartbeater.cc:507] Master 127.10.46.62:36789 requested a full tablet report, sending...
I20260430 20:01:58.276880 14087 ts_manager.cc:194] Registered new tserver with Master: 03851b6ad2d947df83c3fee9abe771da (127.10.46.2:44939)
I20260430 20:01:58.277895 14087 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:45001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:01:58.337553 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:36789
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:01:58.470188 14424 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:01:58.470481 14424 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:01:58.470551 14424 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:01:58.474552 14424 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:01:58.474642 14424 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:01:58.474781 14424 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:01:58.479632 14424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:01:58.481158 14424 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:01:58.482316 14424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:01:58.490131 14430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:58.490131 14429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:01:58.490131 14432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:01:58.490483 14424 server_base.cc:1061] running on GCE node
I20260430 20:01:58.490993 14424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:01:58.491560 14424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:01:58.492820 14424 hybrid_clock.cc:648] HybridClock initialized: now 1777579318492808 us; error 36 us; skew 500 ppm
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:01:58.496879 14424 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:01:58.498289 14424 webserver.cc:492] Webserver started at http://127.10.46.3:42075/ using document root <none> and password file <none>
I20260430 20:01:58.499051 14424 fs_manager.cc:362] Metadata directory not provided
I20260430 20:01:58.499137 14424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:01:58.499388 14424 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:01:58.501390 14424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "49964fe1253340ce85492ac7cd7a9b67"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "66ab26a42df640d569909c0bc1d7313f"
server_key_iv: "941249500f2ea21032613313a7526110"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.501993 14424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "49964fe1253340ce85492ac7cd7a9b67"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "66ab26a42df640d569909c0bc1d7313f"
server_key_iv: "941249500f2ea21032613313a7526110"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.505923 14424 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260430 20:01:58.508826 14439 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.510711 14424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:01:58.510859 14424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "49964fe1253340ce85492ac7cd7a9b67"
format_stamp: "Formatted at 2026-04-30 20:01:58 on dist-test-slave-0bdw"
server_key: "66ab26a42df640d569909c0bc1d7313f"
server_key_iv: "941249500f2ea21032613313a7526110"
server_key_version: "encryptionkey@0"
I20260430 20:01:58.511018 14424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:01:58.528218 14424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:01:58.532620 14424 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:01:58.532825 14424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:01:58.533418 14424 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:01:58.534441 14424 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:01:58.534499 14424 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.534543 14424 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:01:58.534557 14424 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:01:58.548158 14424 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:44417
I20260430 20:01:58.548173 14552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:44417 every 8 connection(s)
I20260430 20:01:58.549435 14424 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:01:58.556545 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14424
I20260430 20:01:58.556697 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:01:58.557008 10424 external_mini_cluster.cc:1468] Setting key 4c810c8e07dc6aff43bab621ebfd1b15
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:58.570487 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:58.552717 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:53911 (local address 127.10.46.62:36789)
0430 20:01:58.552952 (+ 235us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:58.552956 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:58.554131 (+ 1175us) server_negotiation.cc:408] Connection header received
0430 20:01:58.555085 (+ 954us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:58.555089 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:58.555151 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:58.555270 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:58.556995 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.557622 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:58.558388 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.558560 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:58.563969 (+ 5409us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:58.563999 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:58.564002 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:58.564056 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:58.566866 (+ 2810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.568088 (+ 1222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.568092 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.568094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.568165 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.568538 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.568541 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.568601 (+ 60us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.568782 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:58.569002 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:58.570120 (+ 1118us) server_negotiation.cc:300] Negotiation successful
0430 20:01:58.570304 (+ 184us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260430 20:01:58.572379 14555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:58.553420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:36789 (local address 127.10.46.3:53911)
0430 20:01:58.553952 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0430 20:01:58.553992 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 20:01:58.554862 (+ 870us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:01:58.555434 (+ 572us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:01:58.555442 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:01:58.555976 (+ 534us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:01:58.556784 (+ 808us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:58.556802 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.557760 (+ 958us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:01:58.557763 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:01:58.558257 (+ 494us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:01:58.558265 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.558508 (+ 243us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:58.560236 (+ 1728us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:01:58.560276 (+ 40us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:01:58.563538 (+ 3262us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:01:58.567277 (+ 3739us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:58.567294 (+ 17us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:58.567311 (+ 17us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:58.567712 (+ 401us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:58.568295 (+ 583us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:01:58.568298 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:01:58.568301 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:01:58.568418 (+ 117us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:01:58.569026 (+ 608us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:01:58.569033 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:01:58.569300 (+ 267us) client_negotiation.cc:770] Sending connection context
0430 20:01:58.569843 (+ 543us) client_negotiation.cc:241] Negotiation successful
0430 20:01:58.570319 (+ 476us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":138,"threads_started":1}
I20260430 20:01:58.575006 14553 heartbeater.cc:344] Connected to a master server at 127.10.46.62:36789
I20260430 20:01:58.575557 14553 heartbeater.cc:461] Registering TS with master...
I20260430 20:01:58.576880 14553 heartbeater.cc:507] Master 127.10.46.62:36789 requested a full tablet report, sending...
I20260430 20:01:58.579072 14087 ts_manager.cc:194] Registered new tserver with Master: 49964fe1253340ce85492ac7cd7a9b67 (127.10.46.3:44417)
I20260430 20:01:58.579885 14087 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:53911
I20260430 20:01:58.588425 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:01:58.622268 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:01:58.608905 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.10.46.62:36789)
0430 20:01:58.609151 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0430 20:01:58.609156 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:01:58.609265 (+ 109us) server_negotiation.cc:408] Connection header received
0430 20:01:58.609455 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:01:58.609459 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:01:58.609525 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:01:58.609801 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:01:58.611215 (+ 1414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.612301 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:01:58.613857 (+ 1556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:01:58.614116 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:01:58.617678 (+ 3562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:01:58.617712 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:01:58.617715 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:01:58.617787 (+ 72us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:01:58.619629 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.620398 (+ 769us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.620402 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.620403 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.620458 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:01:58.620934 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:01:58.620939 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:01:58.620941 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:01:58.621243 (+ 302us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:01:58.621439 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0430 20:01:58.621956 (+ 517us) server_negotiation.cc:300] Negotiation successful
0430 20:01:58.622120 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260430 20:01:58.622916 14087 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41422
I20260430 20:01:58.624084 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14152
I20260430 20:01:58.631114 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14288
I20260430 20:01:58.641343 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14424
I20260430 20:01:58.648015 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14056
2026-04-30T20:01:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3662 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14565](info): set up 2 sockets
Apr 30 20:01:58 dist-test-slave-0bdw krb5kdc[14565](info): commencing operation
krb5kdc: starting...
W20260430 20:02:00.858903 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.180s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:00 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:01.064563 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:40343
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:01.206095 14581 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:01.206434 14581 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:01.206527 14581 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:01.210268 14581 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:01.210654 14581 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:01.210815 14581 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:01.210891 14581 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:01.210937 14581 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:01.217664 14581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:40343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:01.219731 14581 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:01.221292 14581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 20:02:01.229581 14581 server_base.cc:1061] running on GCE node
W20260430 20:02:01.229475 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:01.229523 14589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:01.229475 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:01.231434 14581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:01.232746 14581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:01.236197 14581 hybrid_clock.cc:648] HybridClock initialized: now 1777579321236084 us; error 153 us; skew 500 ppm
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:01.240748 14581 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:02:01.242233 14581 webserver.cc:492] Webserver started at http://127.10.46.62:39997/ using document root <none> and password file <none>
I20260430 20:02:01.242918 14581 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:01.243010 14581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:01.243243 14581 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:01.245254 14581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "b5f50fde86454ef095b6709f86c18286"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "a488e3c52dad9c8792a0eb1f03cd5833"
server_key_iv: "d0a1928b7da3abdd73fb4787b8dde1f7"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.245968 14581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "b5f50fde86454ef095b6709f86c18286"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "a488e3c52dad9c8792a0eb1f03cd5833"
server_key_iv: "d0a1928b7da3abdd73fb4787b8dde1f7"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.250631 14581 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:02:01.254104 14596 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.255976 14581 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260430 20:02:01.256155 14581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "b5f50fde86454ef095b6709f86c18286"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "a488e3c52dad9c8792a0eb1f03cd5833"
server_key_iv: "d0a1928b7da3abdd73fb4787b8dde1f7"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.256299 14581 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:01.267514 14581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:01.275998 14581 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:01.276304 14581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:01.285542 14581 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:40343
I20260430 20:02:01.285596 14648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:40343 every 8 connection(s)
I20260430 20:02:01.286759 14581 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:01.290570 14649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:01.293336 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14581
I20260430 20:02:01.293587 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:01.293896 10424 external_mini_cluster.cc:1468] Setting key 8ea2c9ef0787b6adb88ac13529e77219
I20260430 20:02:01.300243 14649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286: Bootstrap starting.
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:01.304775 14649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:01.306141 14649 log.cc:826] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:01.308756 14649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286: No bootstrap required, opened a new log
I20260430 20:02:01.311218 14652 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:01.295660 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.10.46.62:40343)
0430 20:02:01.296227 (+ 567us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:01.296237 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:01.296272 (+ 35us) server_negotiation.cc:408] Connection header received
0430 20:02:01.296898 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:01.296918 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:01.297256 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:01.297636 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:01.298611 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.299509 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:01.300607 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.301004 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:01.304504 (+ 3500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:01.304572 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:01.304599 (+ 27us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:01.304693 (+ 94us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:01.307277 (+ 2584us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.307834 (+ 557us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.307840 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.307846 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.307943 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.308257 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.308261 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.308263 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.308686 (+ 423us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:01.308879 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:01.309386 (+ 507us) server_negotiation.cc:300] Negotiation successful
0430 20:02:01.310075 (+ 689us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":199,"threads_started":1}
I20260430 20:02:01.313283 14649 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } }
I20260430 20:02:01.313580 14649 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:01.313640 14649 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f50fde86454ef095b6709f86c18286, State: Initialized, Role: FOLLOWER
I20260430 20:02:01.314090 14649 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } }
I20260430 20:02:01.314216 14649 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:01.314255 14649 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:01.314321 14649 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:01.315460 14649 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } }
I20260430 20:02:01.315954 14649 leader_election.cc:304] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f50fde86454ef095b6709f86c18286; no voters:
I20260430 20:02:01.316319 14649 leader_election.cc:290] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:01.316463 14654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:01.316742 14654 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [term 1 LEADER]: Becoming Leader. State: Replica: b5f50fde86454ef095b6709f86c18286, State: Running, Role: LEADER
I20260430 20:02:01.317042 14654 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } }
I20260430 20:02:01.318157 14649 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:01.319214 14656 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f50fde86454ef095b6709f86c18286. Latest consensus state: current_term: 1 leader_uuid: "b5f50fde86454ef095b6709f86c18286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } } }
I20260430 20:02:01.319366 14656 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:01.319746 14655 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f50fde86454ef095b6709f86c18286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f50fde86454ef095b6709f86c18286" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 40343 } } }
I20260430 20:02:01.319936 14655 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:01.320683 14662 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:01.323922 14662 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:01.330245 14662 catalog_manager.cc:1357] Generated new cluster ID: a56dcc2c3ae54f77aa8d463eb55f0a4c
I20260430 20:02:01.330350 14662 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:01.352404 14662 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:01.353684 14662 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:01.365264 14662 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b5f50fde86454ef095b6709f86c18286: Generated new TSK 0
I20260430 20:02:01.366318 14662 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:01.457605 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:40343
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:01.582698 14677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:01.582953 14677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:01.583014 14677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:01.587049 14677 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:01.587149 14677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:01.587239 14677 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:01.591962 14677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:40343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:01.593196 14677 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:01.594305 14677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:01.607131 14682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:01.607138 14683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:01.607352 14677 server_base.cc:1061] running on GCE node
W20260430 20:02:01.607597 14685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:01.608472 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:01.609380 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:01.610737 14677 hybrid_clock.cc:648] HybridClock initialized: now 1777579321610706 us; error 62 us; skew 500 ppm
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:01.617048 14677 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:02:01.620342 14677 webserver.cc:492] Webserver started at http://127.10.46.1:37367/ using document root <none> and password file <none>
I20260430 20:02:01.621430 14677 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:01.621816 14677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:01.622188 14677 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:01.624818 14677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "db3243646be347259bfbd9918d241ff3"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "526471e637c687862b60ca51ed20d256"
server_key_iv: "1f8853a94eeadcaac6a67093ec302e47"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.625401 14677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "db3243646be347259bfbd9918d241ff3"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "526471e637c687862b60ca51ed20d256"
server_key_iv: "1f8853a94eeadcaac6a67093ec302e47"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.632092 14677 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260430 20:02:01.635174 14692 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.636662 14677 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:01.636897 14677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "db3243646be347259bfbd9918d241ff3"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "526471e637c687862b60ca51ed20d256"
server_key_iv: "1f8853a94eeadcaac6a67093ec302e47"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.637049 14677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:01.648865 14677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:01.653052 14677 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:01.653330 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:01.654098 14677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:01.655541 14677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:01.655676 14677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.655799 14677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:01.655858 14677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260430 20:02:01.670466 14677 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:40009
I20260430 20:02:01.670568 14805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:40009 every 8 connection(s)
I20260430 20:02:01.672497 14677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:01.676481 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14677
I20260430 20:02:01.676646 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:01.676898 10424 external_mini_cluster.cc:1468] Setting key 784e5bcc1decadac014ae07bc70af87c
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:02:01.687589 14652 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:01.674676 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:43901 (local address 127.10.46.62:40343)
0430 20:02:01.674860 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:01.674864 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:01.675638 (+ 774us) server_negotiation.cc:408] Connection header received
0430 20:02:01.676704 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:01.676708 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:01.676776 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:01.676895 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:01.678614 (+ 1719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.679146 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:01.679985 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.680140 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:01.682873 (+ 2733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:01.682905 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:01.682907 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:01.682938 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:01.684930 (+ 1992us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.685608 (+ 678us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.685612 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.685614 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.685687 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.686139 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.686146 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.686151 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.686358 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:01.686454 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:01.687285 (+ 831us) server_negotiation.cc:300] Negotiation successful
0430 20:02:01.687440 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:02:01.688441 14808 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:01.674984 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:40343 (local address 127.10.46.1:43901)
0430 20:02:01.675467 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:01.675503 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:01.676461 (+ 958us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:01.677102 (+ 641us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:01.677111 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:01.677522 (+ 411us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:01.678422 (+ 900us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:01.678435 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.679296 (+ 861us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:01.679302 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:01.679830 (+ 528us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:01.679843 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.680106 (+ 263us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:01.680737 (+ 631us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:01.680767 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:01.682662 (+ 1895us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:01.685134 (+ 2472us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:01.685141 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:01.685156 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:01.685463 (+ 307us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:01.685801 (+ 338us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:01.685805 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:01.685809 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:01.685985 (+ 176us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:01.686474 (+ 489us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:01.686480 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:01.686931 (+ 451us) client_negotiation.cc:770] Sending connection context
0430 20:02:01.687184 (+ 253us) client_negotiation.cc:241] Negotiation successful
0430 20:02:01.687517 (+ 333us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:02:01.690331 14806 heartbeater.cc:344] Connected to a master server at 127.10.46.62:40343
I20260430 20:02:01.690900 14806 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:01.691713 14806 heartbeater.cc:507] Master 127.10.46.62:40343 requested a full tablet report, sending...
I20260430 20:02:01.694432 14613 ts_manager.cc:194] Registered new tserver with Master: db3243646be347259bfbd9918d241ff3 (127.10.46.1:40009)
I20260430 20:02:01.695703 14613 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:43901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:01.745414 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:40343
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:01.867072 14813 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:01.867357 14813 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:01.867580 14813 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:01.873626 14813 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:01.873778 14813 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:01.874097 14813 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:01.879662 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:40343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:01.880981 14813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:01.881884 14813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:01.889077 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:01.889077 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:01.889434 14813 server_base.cc:1061] running on GCE node
W20260430 20:02:01.889107 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:01.890352 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:01.891130 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:01.892369 14813 hybrid_clock.cc:648] HybridClock initialized: now 1777579321892348 us; error 35 us; skew 500 ppm
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:01.896466 14813 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:02:01.898653 14813 webserver.cc:492] Webserver started at http://127.10.46.2:37483/ using document root <none> and password file <none>
I20260430 20:02:01.899425 14813 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:01.899489 14813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:01.899726 14813 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:01.901607 14813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "8173aa1fb71147b1aae13fe8fd9ed0fd"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "139f914863e8b7939511d5130e4ffb0e"
server_key_iv: "2d56953a45222ba2c5e29f8b86db7c6a"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.902117 14813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "8173aa1fb71147b1aae13fe8fd9ed0fd"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "139f914863e8b7939511d5130e4ffb0e"
server_key_iv: "2d56953a45222ba2c5e29f8b86db7c6a"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.906431 14813 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:02:01.909497 14828 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.910908 14813 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:01.911101 14813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "8173aa1fb71147b1aae13fe8fd9ed0fd"
format_stamp: "Formatted at 2026-04-30 20:02:01 on dist-test-slave-0bdw"
server_key: "139f914863e8b7939511d5130e4ffb0e"
server_key_iv: "2d56953a45222ba2c5e29f8b86db7c6a"
server_key_version: "encryptionkey@0"
I20260430 20:02:01.911242 14813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:01.955544 14813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:01.959383 14813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:01.959729 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:01.960487 14813 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:01.961807 14813 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:01.961977 14813 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.962065 14813 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:01.962123 14813 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:01.980240 14813 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:33435
I20260430 20:02:01.980257 14941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:33435 every 8 connection(s)
I20260430 20:02:01.981325 14813 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:01.984184 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14813
I20260430 20:02:01.984323 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:02:01.984582 10424 external_mini_cluster.cc:1468] Setting key 39b5bb6249c29db9bf3bff392465d124
Apr 30 20:02:01 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
I20260430 20:02:02.000882 14652 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:01.983548 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:50915 (local address 127.10.46.62:40343)
0430 20:02:01.983817 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:01.983824 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:01.984548 (+ 724us) server_negotiation.cc:408] Connection header received
0430 20:02:01.985528 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:01.985531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:01.985588 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:01.985688 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:01.988132 (+ 2444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.988752 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:01.990118 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.990359 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:01.994861 (+ 4502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:01.994889 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:01.994892 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:01.994932 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:01.997592 (+ 2660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.998311 (+ 719us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.998315 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.998317 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.998371 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:01.998906 (+ 535us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:01.998910 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:01.998912 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:01.999352 (+ 440us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:01.999725 (+ 373us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:02.000543 (+ 818us) server_negotiation.cc:300] Negotiation successful
0430 20:02:02.000698 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260430 20:02:02.001564 14944 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:01.983528 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:40343 (local address 127.10.46.2:50915)
0430 20:02:01.984115 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:01.984420 (+ 305us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:01.985318 (+ 898us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:01.985859 (+ 541us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:01.985868 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:01.986291 (+ 423us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:01.987619 (+ 1328us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:01.987651 (+ 32us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.989061 (+ 1410us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:01.989066 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:01.989798 (+ 732us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:01.989821 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:01.990250 (+ 429us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:01.991707 (+ 1457us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:01.991747 (+ 40us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:01.994552 (+ 2805us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:01.997838 (+ 3286us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:01.997845 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:01.997858 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:01.998167 (+ 309us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:01.998482 (+ 315us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:01.998491 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:01.998492 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:01.998744 (+ 252us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:01.999759 (+ 1015us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:01.999904 (+ 145us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:02.000219 (+ 315us) client_negotiation.cc:770] Sending connection context
0430 20:02:02.000469 (+ 250us) client_negotiation.cc:241] Negotiation successful
0430 20:02:02.000787 (+ 318us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":171,"threads_started":1}
I20260430 20:02:02.003997 14942 heartbeater.cc:344] Connected to a master server at 127.10.46.62:40343
I20260430 20:02:02.004477 14942 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:02.005438 14942 heartbeater.cc:507] Master 127.10.46.62:40343 requested a full tablet report, sending...
I20260430 20:02:02.007853 14613 ts_manager.cc:194] Registered new tserver with Master: 8173aa1fb71147b1aae13fe8fd9ed0fd (127.10.46.2:33435)
I20260430 20:02:02.008621 14613 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:50915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:02.056413 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:40343
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:02.174103 14949 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:02.174355 14949 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:02.174415 14949 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:02.177986 14949 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:02.178079 14949 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:02.178164 14949 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:02.182919 14949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:40343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:02.184123 14949 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:02.184976 14949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:02.193602 14954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:02.193568 14957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:02.193888 14955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:02.195922 14949 server_base.cc:1061] running on GCE node
I20260430 20:02:02.197150 14949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:02.199536 14949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:02.201016 14949 hybrid_clock.cc:648] HybridClock initialized: now 1777579322200897 us; error 142 us; skew 500 ppm
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:02.205672 14949 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:02:02.207193 14949 webserver.cc:492] Webserver started at http://127.10.46.3:38357/ using document root <none> and password file <none>
I20260430 20:02:02.207921 14949 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:02.208020 14949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:02.208546 14949 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:02.211706 14949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "0a2c845e0d4b4aa9959e2eee12b68b12"
format_stamp: "Formatted at 2026-04-30 20:02:02 on dist-test-slave-0bdw"
server_key: "6ab89ae349e5cca2b1cd2d55e292667b"
server_key_iv: "d7d9ce464e41817f839e23ec774acf9a"
server_key_version: "encryptionkey@0"
I20260430 20:02:02.212854 14949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "0a2c845e0d4b4aa9959e2eee12b68b12"
format_stamp: "Formatted at 2026-04-30 20:02:02 on dist-test-slave-0bdw"
server_key: "6ab89ae349e5cca2b1cd2d55e292667b"
server_key_iv: "d7d9ce464e41817f839e23ec774acf9a"
server_key_version: "encryptionkey@0"
I20260430 20:02:02.217438 14949 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:02.223428 14964 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:02.225030 14949 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:02.225256 14949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "0a2c845e0d4b4aa9959e2eee12b68b12"
format_stamp: "Formatted at 2026-04-30 20:02:02 on dist-test-slave-0bdw"
server_key: "6ab89ae349e5cca2b1cd2d55e292667b"
server_key_iv: "d7d9ce464e41817f839e23ec774acf9a"
server_key_version: "encryptionkey@0"
I20260430 20:02:02.225405 14949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:02.242120 14949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:02.246282 14949 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:02.246520 14949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:02.247149 14949 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:02.248236 14949 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:02.248319 14949 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:02.248389 14949 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:02.248440 14949 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:02.263038 14949 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:42883
I20260430 20:02:02.263059 15077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:42883 every 8 connection(s)
I20260430 20:02:02.264261 14949 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:02.274482 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 14949
I20260430 20:02:02.274632 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:02.274891 10424 external_mini_cluster.cc:1468] Setting key 4092b0c963cfe6889be7077fc8b84c51
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:02.280153 14652 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:02.266527 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:34669 (local address 127.10.46.62:40343)
0430 20:02:02.266791 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:02.266796 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:02.267378 (+ 582us) server_negotiation.cc:408] Connection header received
0430 20:02:02.268772 (+ 1394us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:02.268777 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:02.269036 (+ 259us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:02.269205 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:02.271573 (+ 2368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.272195 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:02.272928 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.273104 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:02.276006 (+ 2902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:02.276024 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:02.276032 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:02.276063 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:02.277678 (+ 1615us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.278209 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.278213 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.278215 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.278266 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.278601 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.278604 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.278606 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.279102 (+ 496us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:02.279243 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:02.279832 (+ 589us) server_negotiation.cc:300] Negotiation successful
0430 20:02:02.279980 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260430 20:02:02.280833 15080 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:02.266687 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:40343 (local address 127.10.46.3:34669)
0430 20:02:02.267177 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:02.267212 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:02.268381 (+ 1169us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:02.269452 (+ 1071us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:02.269465 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:02.270307 (+ 842us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:02.271288 (+ 981us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:02.271307 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.272350 (+ 1043us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:02.272355 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:02.272780 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:02.272788 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.273069 (+ 281us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:02.273794 (+ 725us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:02.273816 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:02.275804 (+ 1988us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:02.277812 (+ 2008us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:02.277819 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:02.277832 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:02.278092 (+ 260us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:02.278370 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:02.278378 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:02.278379 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:02.278487 (+ 108us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:02.279243 (+ 756us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:02.279251 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:02.279499 (+ 248us) client_negotiation.cc:770] Sending connection context
0430 20:02:02.279752 (+ 253us) client_negotiation.cc:241] Negotiation successful
0430 20:02:02.280036 (+ 284us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":113,"threads_started":1}
I20260430 20:02:02.282243 15078 heartbeater.cc:344] Connected to a master server at 127.10.46.62:40343
I20260430 20:02:02.282559 15078 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:02.283138 15078 heartbeater.cc:507] Master 127.10.46.62:40343 requested a full tablet report, sending...
I20260430 20:02:02.284437 14613 ts_manager.cc:194] Registered new tserver with Master: 0a2c845e0d4b4aa9959e2eee12b68b12 (127.10.46.3:42883)
I20260430 20:02:02.285125 14613 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:34669
I20260430 20:02:02.291529 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:02:02.324115 15085 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:02.311862 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.10.46.1:40009)
0430 20:02:02.312360 (+ 498us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:02.312368 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:02.312504 (+ 136us) server_negotiation.cc:408] Connection header received
0430 20:02:02.312600 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:02.312605 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:02.312792 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:02.312918 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:02.313844 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.314580 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:02.315945 (+ 1365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.316193 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:02.318803 (+ 2610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:02.318835 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:02.318848 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:02.318883 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:02.321519 (+ 2636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.322178 (+ 659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.322184 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.322186 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.322244 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.322583 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.322587 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.322588 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.322819 (+ 231us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:02.322987 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:02.323588 (+ 601us) server_negotiation.cc:300] Negotiation successful
0430 20:02:02.323882 (+ 294us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":104,"threads_started":1}
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:02.339730 14652 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:02.327743 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.10.46.62:40343)
0430 20:02:02.328298 (+ 555us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:02.328302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:02.328324 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:02:02.328387 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:02.328390 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:02.328457 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:02.328640 (+ 183us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:02.329726 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.330643 (+ 917us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:02.331729 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:02.332023 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:02.334818 (+ 2795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:02.334846 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:02.334848 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:02.334887 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:02.336890 (+ 2003us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.337738 (+ 848us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.337742 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.337744 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.337821 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:02.338255 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:02.338258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:02.338259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:02.338469 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:02.338595 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:02.339106 (+ 511us) server_negotiation.cc:300] Negotiation successful
0430 20:02:02.339510 (+ 404us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260430 20:02:02.340390 14613 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37094
I20260430 20:02:02.341439 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14677
I20260430 20:02:02.348546 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14813
I20260430 20:02:02.355638 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14949
I20260430 20:02:02.362360 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 14581
2026-04-30T20:02:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3710 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[15090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[15090](info): set up 2 sockets
Apr 30 20:02:02 dist-test-slave-0bdw krb5kdc[15090](info): commencing operation
krb5kdc: starting...
W20260430 20:02:04.497677 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.108s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:04 dist-test-slave-0bdw krb5kdc[15090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:04.689380 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:46463
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:46463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:04.815860 15106 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:04.816128 15106 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:04.816190 15106 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:04.819658 15106 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:04.819756 15106 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:04.819810 15106 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:04.819831 15106 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:04.819849 15106 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:04.824748 15106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:46463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:04.825946 15106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:04.826783 15106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:04.832717 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:04.832867 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:04.833230 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:04.833195 15106 server_base.cc:1061] running on GCE node
I20260430 20:02:04.833817 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:04.834796 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:04.836117 15106 hybrid_clock.cc:648] HybridClock initialized: now 1777579324836009 us; error 139 us; skew 500 ppm
Apr 30 20:02:04 dist-test-slave-0bdw krb5kdc[15090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:04.839139 15106 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:02:04.840088 15106 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:04.840344 15106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:04.840631 15106 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:04.842913 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "f92472a2adde440ab221a7815921bca1"
format_stamp: "Formatted at 2026-04-30 20:02:04 on dist-test-slave-0bdw"
server_key: "c0afb36a0b3cad58b97afe68ab38f650"
server_key_iv: "d5717d8a6e6aceb41588425525fd563f"
server_key_version: "encryptionkey@0"
I20260430 20:02:04.843448 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "f92472a2adde440ab221a7815921bca1"
format_stamp: "Formatted at 2026-04-30 20:02:04 on dist-test-slave-0bdw"
server_key: "c0afb36a0b3cad58b97afe68ab38f650"
server_key_iv: "d5717d8a6e6aceb41588425525fd563f"
server_key_version: "encryptionkey@0"
I20260430 20:02:04.848922 15106 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260430 20:02:04.852721 15120 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:04.854055 15106 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:04.854234 15106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "f92472a2adde440ab221a7815921bca1"
format_stamp: "Formatted at 2026-04-30 20:02:04 on dist-test-slave-0bdw"
server_key: "c0afb36a0b3cad58b97afe68ab38f650"
server_key_iv: "d5717d8a6e6aceb41588425525fd563f"
server_key_version: "encryptionkey@0"
I20260430 20:02:04.854367 15106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:04.874459 15106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:04.878119 15106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:04.878360 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:04.887724 15106 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:46463
I20260430 20:02:04.887739 15172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:46463 every 8 connection(s)
I20260430 20:02:04.888576 15106 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:04.891999 15173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:04.899209 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15106
I20260430 20:02:04.899343 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:04.899569 10424 external_mini_cluster.cc:1468] Setting key ea859940211687729350d4428112dc7a
I20260430 20:02:04.902318 15173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: Bootstrap starting.
I20260430 20:02:04.905089 15173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:04.906136 15173 log.cc:826] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:04.909677 15173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: No bootstrap required, opened a new log
Apr 30 20:02:04 dist-test-slave-0bdw krb5kdc[15090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:04.915021 15173 raft_consensus.cc:359] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } }
I20260430 20:02:04.915313 15173 raft_consensus.cc:385] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:04.915371 15173 raft_consensus.cc:740] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92472a2adde440ab221a7815921bca1, State: Initialized, Role: FOLLOWER
I20260430 20:02:04.915969 15173 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } }
I20260430 20:02:04.916116 15173 raft_consensus.cc:399] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:04.916204 15173 raft_consensus.cc:493] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:04.916306 15173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:04.917865 15173 raft_consensus.cc:515] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } }
I20260430 20:02:04.918102 15176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:04.901226 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.10.46.62:46463)
0430 20:02:04.901970 (+ 744us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:04.901991 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:04.902057 (+ 66us) server_negotiation.cc:408] Connection header received
0430 20:02:04.903187 (+ 1130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:04.903217 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:04.903950 (+ 733us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:04.904370 (+ 420us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:04.905443 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:04.906609 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:04.907970 (+ 1361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:04.908530 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:04.911114 (+ 2584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:04.911143 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:04.911161 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:04.911299 (+ 138us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:04.914669 (+ 3370us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:04.915276 (+ 607us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:04.915284 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:04.915291 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:04.915401 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:04.915844 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:04.915848 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:04.915850 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:04.916541 (+ 691us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:04.916759 (+ 218us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:04.917115 (+ 356us) server_negotiation.cc:300] Negotiation successful
0430 20:02:04.917382 (+ 267us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":172,"threads_started":1}
I20260430 20:02:04.918325 15173 leader_election.cc:304] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f92472a2adde440ab221a7815921bca1; no voters:
I20260430 20:02:04.918725 15173 leader_election.cc:290] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:04.918957 15178 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:04.919230 15178 raft_consensus.cc:697] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [term 1 LEADER]: Becoming Leader. State: Replica: f92472a2adde440ab221a7815921bca1, State: Running, Role: LEADER
I20260430 20:02:04.919971 15178 consensus_queue.cc:237] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } }
I20260430 20:02:04.920830 15173 sys_catalog.cc:565] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:04.922457 15179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f92472a2adde440ab221a7815921bca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } } }
I20260430 20:02:04.922636 15179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:04.923049 15180 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92472a2adde440ab221a7815921bca1. Latest consensus state: current_term: 1 leader_uuid: "f92472a2adde440ab221a7815921bca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92472a2adde440ab221a7815921bca1" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 46463 } } }
I20260430 20:02:04.923197 15180 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:04.924249 15187 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:04.927279 15187 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:04.933552 15187 catalog_manager.cc:1357] Generated new cluster ID: 63eba80ecfa84220b045590c3d02a199
I20260430 20:02:04.933671 15187 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:04.941461 15187 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:04.942750 15187 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:04.954834 15187 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: Generated new TSK 0
I20260430 20:02:04.956048 15187 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:05.025440 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:46463
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:05.160630 15201 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:05.160897 15201 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:05.160970 15201 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:05.165198 15201 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:05.165295 15201 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:05.165385 15201 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:05.171254 15201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:05.173386 15201 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:05.174765 15201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:05.183115 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:05.183193 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:05.183522 15201 server_base.cc:1061] running on GCE node
W20260430 20:02:05.183176 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:05.184233 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:05.184902 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:05.186233 15201 hybrid_clock.cc:648] HybridClock initialized: now 1777579325186209 us; error 41 us; skew 500 ppm
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:05.189857 15201 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:02:05.190689 15201 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:05.190754 15201 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:05.190954 15201 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:05.193857 15201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "8835c3a71d6843edb9d2911616d89e4e"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "761e2277356f980f5d86a4adb35ff0ea"
server_key_iv: "2031cc60649e55834fb13dc1ce33371f"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.194449 15201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "8835c3a71d6843edb9d2911616d89e4e"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "761e2277356f980f5d86a4adb35ff0ea"
server_key_iv: "2031cc60649e55834fb13dc1ce33371f"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.200508 15201 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260430 20:02:05.203205 15215 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.204849 15201 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:05.205005 15201 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "8835c3a71d6843edb9d2911616d89e4e"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "761e2277356f980f5d86a4adb35ff0ea"
server_key_iv: "2031cc60649e55834fb13dc1ce33371f"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.205117 15201 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:05.229518 15201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:05.234124 15201 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:05.234315 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:05.235085 15201 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:05.236223 15201 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:05.236280 15201 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.236325 15201 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:05.236339 15201 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.249173 15201 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:39603
I20260430 20:02:05.249223 15328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:39603 every 8 connection(s)
I20260430 20:02:05.249819 15201 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:05.254689 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15201
I20260430 20:02:05.254907 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:05.255291 10424 external_mini_cluster.cc:1468] Setting key 5c34085d1f45b22577ac8e879975dac0
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:02:05.268074 15176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.252084 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:56833 (local address 127.10.46.62:46463)
0430 20:02:05.252271 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.252275 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.253270 (+ 995us) server_negotiation.cc:408] Connection header received
0430 20:02:05.254750 (+ 1480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.254754 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.254821 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.254937 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:05.256300 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.257282 (+ 982us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.258384 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.258659 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.262478 (+ 3819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:05.262514 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:05.262521 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:05.262565 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:05.265000 (+ 2435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.265715 (+ 715us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.265723 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.265726 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.265791 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.266292 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.266296 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.266298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.266538 (+ 240us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:05.266664 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.267602 (+ 938us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.267836 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260430 20:02:05.268498 15331 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.252529 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:46463 (local address 127.10.46.1:56833)
0430 20:02:05.253089 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.253126 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.254494 (+ 1368us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.255115 (+ 621us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.255124 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.255569 (+ 445us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:05.256150 (+ 581us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.256162 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.257619 (+ 1457us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.257626 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.258205 (+ 579us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.258216 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.258609 (+ 393us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.259589 (+ 980us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:05.259675 (+ 86us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:05.262063 (+ 2388us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:05.265226 (+ 3163us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.265234 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.265246 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.265546 (+ 300us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.265959 (+ 413us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.265970 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.265973 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.266162 (+ 189us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.266758 (+ 596us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:05.266764 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:05.267026 (+ 262us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.267229 (+ 203us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.267450 (+ 221us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":160,"threads_started":1}
I20260430 20:02:05.270373 15329 heartbeater.cc:344] Connected to a master server at 127.10.46.62:46463
I20260430 20:02:05.270629 15329 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:02:05.271260 15329 heartbeater.cc:507] Master 127.10.46.62:46463 requested a full tablet report, sending...
I20260430 20:02:05.273279 15137 ts_manager.cc:194] Registered new tserver with Master: 8835c3a71d6843edb9d2911616d89e4e (127.10.46.1:39603)
I20260430 20:02:05.275007 15137 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:56833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:05.326313 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:46463
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:05.460773 15336 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:05.461041 15336 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:05.461107 15336 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:05.464972 15336 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:05.465073 15336 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:05.465204 15336 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:05.470501 15336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:05.471946 15336 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:05.472937 15336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:05.480331 15341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:05.480538 15344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:05.480679 15342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:05.481278 15336 server_base.cc:1061] running on GCE node
I20260430 20:02:05.481751 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:05.482373 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:05.483691 15336 hybrid_clock.cc:648] HybridClock initialized: now 1777579325483631 us; error 60 us; skew 500 ppm
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:05.486967 15336 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:02:05.488062 15336 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:05.488161 15336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:05.488408 15336 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:05.490877 15336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "99efd6a2a7a8423d861ef99d6c30ebfc"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "3a7695437840c23a31b7c27f9cca46f9"
server_key_iv: "15f2093bce93d971706527cf7f474f6d"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.492203 15336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "99efd6a2a7a8423d861ef99d6c30ebfc"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "3a7695437840c23a31b7c27f9cca46f9"
server_key_iv: "15f2093bce93d971706527cf7f474f6d"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.498462 15336 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260430 20:02:05.501900 15350 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.503420 15336 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:05.503575 15336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "99efd6a2a7a8423d861ef99d6c30ebfc"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "3a7695437840c23a31b7c27f9cca46f9"
server_key_iv: "15f2093bce93d971706527cf7f474f6d"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.503695 15336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:05.522272 15336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:05.527372 15336 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:05.527592 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:05.528312 15336 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:05.529345 15336 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:05.529402 15336 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.529445 15336 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:05.529460 15336 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.545994 15336 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:38311
I20260430 20:02:05.546007 15463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:38311 every 8 connection(s)
I20260430 20:02:05.546730 15336 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:05.555598 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15336
I20260430 20:02:05.555863 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:05.556566 10424 external_mini_cluster.cc:1468] Setting key 105cbf69526ae8101b9de855b6e06cd3
I20260430 20:02:05.563444 15176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.548798 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:45231 (local address 127.10.46.62:46463)
0430 20:02:05.549003 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.549007 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.549737 (+ 730us) server_negotiation.cc:408] Connection header received
0430 20:02:05.551133 (+ 1396us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.551137 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.551202 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.551307 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:05.552697 (+ 1390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.553231 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.553928 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.554111 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.557482 (+ 3371us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:05.557516 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:05.557519 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:05.557560 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:05.559946 (+ 2386us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.560735 (+ 789us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.560738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.560740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.560791 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.561290 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.561295 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.561297 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.561604 (+ 307us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:05.561971 (+ 367us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.563010 (+ 1039us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.563183 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260430 20:02:05.564013 15466 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.549084 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:46463 (local address 127.10.46.2:45231)
0430 20:02:05.549534 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.549573 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.550903 (+ 1330us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.551486 (+ 583us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.551496 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.552026 (+ 530us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:05.552523 (+ 497us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.552534 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.553364 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.553370 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.553791 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.553798 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.554035 (+ 237us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.554700 (+ 665us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:05.554720 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:05.557096 (+ 2376us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:05.560151 (+ 3055us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.560163 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.560182 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.560592 (+ 410us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.560923 (+ 331us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.560929 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.560934 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.561101 (+ 167us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.562003 (+ 902us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:05.562011 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:05.562627 (+ 616us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.562901 (+ 274us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.563151 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260430 20:02:05.565356 15464 heartbeater.cc:344] Connected to a master server at 127.10.46.62:46463
I20260430 20:02:05.565667 15464 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:05.566219 15464 heartbeater.cc:507] Master 127.10.46.62:46463 requested a full tablet report, sending...
I20260430 20:02:05.567620 15137 ts_manager.cc:194] Registered new tserver with Master: 99efd6a2a7a8423d861ef99d6c30ebfc (127.10.46.2:38311)
I20260430 20:02:05.568506 15137 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:45231
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:05.637048 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:46463
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:05.755126 15471 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:05.755384 15471 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:05.755448 15471 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:05.759274 15471 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:05.759434 15471 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:05.759586 15471 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:05.764699 15471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:05.765991 15471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:05.766927 15471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:05.777415 15476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:05.777406 15479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:05.777639 15477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:05.777918 15471 server_base.cc:1061] running on GCE node
I20260430 20:02:05.778692 15471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:05.779378 15471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:05.780628 15471 hybrid_clock.cc:648] HybridClock initialized: now 1777579325780582 us; error 57 us; skew 500 ppm
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:05.784446 15471 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:02:05.785363 15471 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:05.785454 15471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:05.785835 15471 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:05.787760 15471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "5034f483700041e397bd6c0ceca4a21c"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "1e1952648b31676bbac3831741a944a7"
server_key_iv: "9dd6a92da44a9a1be8159e7a489784f6"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.788379 15471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "5034f483700041e397bd6c0ceca4a21c"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "1e1952648b31676bbac3831741a944a7"
server_key_iv: "9dd6a92da44a9a1be8159e7a489784f6"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.792883 15471 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260430 20:02:05.796157 15485 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.798059 15471 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:02:05.798262 15471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "5034f483700041e397bd6c0ceca4a21c"
format_stamp: "Formatted at 2026-04-30 20:02:05 on dist-test-slave-0bdw"
server_key: "1e1952648b31676bbac3831741a944a7"
server_key_iv: "9dd6a92da44a9a1be8159e7a489784f6"
server_key_version: "encryptionkey@0"
I20260430 20:02:05.798393 15471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:05.812826 15471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:05.816684 15471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:05.816921 15471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:05.817561 15471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:05.818687 15471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:05.818758 15471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.818825 15471 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:05.818872 15471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:05.830737 15471 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:39199
I20260430 20:02:05.830749 15598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:39199 every 8 connection(s)
I20260430 20:02:05.831421 15471 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:05.835484 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15471
I20260430 20:02:05.835651 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:05.836026 10424 external_mini_cluster.cc:1468] Setting key 3433784ea11b4d4190e9a93d6b836e8d
Apr 30 20:02:05 dist-test-slave-0bdw krb5kdc[15090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:05.847844 15176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.834482 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:42213 (local address 127.10.46.62:46463)
0430 20:02:05.834640 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.834644 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.836106 (+ 1462us) server_negotiation.cc:408] Connection header received
0430 20:02:05.838236 (+ 2130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.838240 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.838306 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.838395 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:05.839978 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.840499 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.841198 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.841374 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.844063 (+ 2689us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:05.844086 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:05.844088 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:05.844119 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:05.845673 (+ 1554us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.846293 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.846297 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.846299 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.846348 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.846676 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.846679 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.846681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.846850 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:05.846959 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.847473 (+ 514us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.847604 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 20:02:05.848460 15601 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.835065 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:46463 (local address 127.10.46.3:42213)
0430 20:02:05.835835 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.835902 (+ 67us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.837913 (+ 2011us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.838640 (+ 727us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.838653 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.839137 (+ 484us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:05.839753 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.839821 (+ 68us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.840638 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.840641 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.841070 (+ 429us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.841077 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.841321 (+ 244us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.842051 (+ 730us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:05.842077 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:05.843870 (+ 1793us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:05.845860 (+ 1990us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.845867 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.845883 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.846160 (+ 277us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.846457 (+ 297us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:05.846461 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:05.846462 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:05.846571 (+ 109us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:05.846964 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:05.846973 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:05.847221 (+ 248us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.847425 (+ 204us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.847670 (+ 245us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":166,"threads_started":1}
I20260430 20:02:05.850382 15599 heartbeater.cc:344] Connected to a master server at 127.10.46.62:46463
I20260430 20:02:05.850795 15599 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:05.851436 15599 heartbeater.cc:507] Master 127.10.46.62:46463 requested a full tablet report, sending...
I20260430 20:02:05.853027 15136 ts_manager.cc:194] Registered new tserver with Master: 5034f483700041e397bd6c0ceca4a21c (127.10.46.3:39199)
I20260430 20:02:05.853855 15136 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:42213
I20260430 20:02:05.865973 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:02:05.879558 15176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.868976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.10.46.62:46463)
0430 20:02:05.869247 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.869254 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.869377 (+ 123us) server_negotiation.cc:408] Connection header received
0430 20:02:05.869605 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.869609 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.869684 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.869835 (+ 151us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:05.870936 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.871724 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.872798 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.873047 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.874877 (+ 1830us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:05.874904 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:05.874907 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:05.874942 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:05.877208 (+ 2266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.877740 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.877744 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.877746 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.877802 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:05.878058 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:05.878061 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:05.878063 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:05.878276 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:05.878613 (+ 337us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.879200 (+ 587us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.879372 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260430 20:02:05.883884 15136 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:02:05.886548 15136 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:05.904273 15610 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.898152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39603 (local address 127.0.0.1:39644)
0430 20:02:05.899569 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.899607 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.899751 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.900879 (+ 1128us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.900883 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.900918 (+ 35us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:05.901213 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.901221 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.902736 (+ 1515us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.902739 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.903701 (+ 962us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.903708 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.903842 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.903917 (+ 75us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.904013 (+ 96us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.904096 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":91,"threads_started":1}
I20260430 20:02:05.905563 15611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.898397 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.10.46.1:39603)
0430 20:02:05.900419 (+ 2022us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.900426 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.900448 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:02:05.900538 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.900543 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.900722 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.901354 (+ 632us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:05.901490 (+ 136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.902599 (+ 1109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.903853 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.904447 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.904573 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.905305 (+ 732us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.905422 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1877,"thread_start_us":139,"threads_started":1}
I20260430 20:02:05.906015 15615 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.900574 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.10.46.3:39199)
0430 20:02:05.901348 (+ 774us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.901354 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.901374 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:02:05.901455 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.901460 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.901627 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.901768 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:05.902307 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.903431 (+ 1124us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.905044 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.905569 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.905705 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.905773 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.905868 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":139,"threads_started":1}
I20260430 20:02:05.904862 15614 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.900061 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:39199 (local address 127.0.0.1:44368)
0430 20:02:05.901036 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.901051 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.901123 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.901906 (+ 783us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.901910 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.901929 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:05.902178 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.902185 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.903566 (+ 1381us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.903569 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.904441 (+ 872us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.904451 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.904564 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.904580 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.904625 (+ 45us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.904742 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":138,"threads_started":1}
I20260430 20:02:05.906281 15612 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.898548 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:38311 (local address 127.0.0.1:59752)
0430 20:02:05.899970 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:05.899989 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:05.900086 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:05.900776 (+ 690us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:05.900782 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:05.900805 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:05.901134 (+ 329us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.901141 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.903546 (+ 2405us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.903553 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:05.904527 (+ 974us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:05.904537 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.904769 (+ 232us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.904799 (+ 30us) client_negotiation.cc:770] Sending connection context
0430 20:02:05.905176 (+ 377us) client_negotiation.cc:241] Negotiation successful
0430 20:02:05.905243 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":140,"threads_started":1}
I20260430 20:02:05.908062 15613 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:05.898974 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.10.46.2:38311)
0430 20:02:05.900096 (+ 1122us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:05.900268 (+ 172us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:05.900294 (+ 26us) server_negotiation.cc:408] Connection header received
0430 20:02:05.900404 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:05.900409 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:05.900597 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:05.902145 (+ 1548us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:05.902304 (+ 159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.903390 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:05.905341 (+ 1951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:05.905900 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:05.906016 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:05.906084 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 20:02:05.907867 (+ 1783us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":462,"threads_started":1}
I20260430 20:02:05.909216 15262 tablet_service.cc:1511] Processing CreateTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 (DEFAULT_TABLE table=test-table [id=8d2138c93bb94933a6300d01243dfb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:05.910538 15262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f163aa47e4e0b8ed1850cd186d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:05.910171 15533 tablet_service.cc:1511] Processing CreateTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 (DEFAULT_TABLE table=test-table [id=8d2138c93bb94933a6300d01243dfb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:05.911990 15533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f163aa47e4e0b8ed1850cd186d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:05.912744 15398 tablet_service.cc:1511] Processing CreateTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 (DEFAULT_TABLE table=test-table [id=8d2138c93bb94933a6300d01243dfb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:05.914158 15398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f163aa47e4e0b8ed1850cd186d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:05.916862 15616 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Bootstrap starting.
I20260430 20:02:05.919360 15616 tablet_bootstrap.cc:654] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:05.920296 15616 log.cc:826] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:05.921896 15618 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Bootstrap starting.
I20260430 20:02:05.923444 15616 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: No bootstrap required, opened a new log
I20260430 20:02:05.923638 15619 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Bootstrap starting.
I20260430 20:02:05.923728 15616 ts_tablet_manager.cc:1403] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260430 20:02:05.925099 15618 tablet_bootstrap.cc:654] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:05.925745 15619 tablet_bootstrap.cc:654] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:05.926299 15618 log.cc:826] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:05.926460 15619 log.cc:826] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:05.927697 15616 raft_consensus.cc:359] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.927981 15616 raft_consensus.cc:385] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:05.928041 15616 raft_consensus.cc:740] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8835c3a71d6843edb9d2911616d89e4e, State: Initialized, Role: FOLLOWER
I20260430 20:02:05.928532 15616 consensus_queue.cc:260] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.928660 15619 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: No bootstrap required, opened a new log
I20260430 20:02:05.928634 15618 tablet_bootstrap.cc:492] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: No bootstrap required, opened a new log
I20260430 20:02:05.928874 15619 ts_tablet_manager.cc:1403] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:02:05.928896 15618 ts_tablet_manager.cc:1403] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260430 20:02:05.929586 15616 ts_tablet_manager.cc:1434] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260430 20:02:05.930099 15329 heartbeater.cc:499] Master 127.10.46.62:46463 was elected leader, sending a full tablet report...
I20260430 20:02:05.932919 15619 raft_consensus.cc:359] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.933187 15619 raft_consensus.cc:385] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:05.933245 15619 raft_consensus.cc:740] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99efd6a2a7a8423d861ef99d6c30ebfc, State: Initialized, Role: FOLLOWER
I20260430 20:02:05.933717 15619 consensus_queue.cc:260] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.934485 15464 heartbeater.cc:499] Master 127.10.46.62:46463 was elected leader, sending a full tablet report...
I20260430 20:02:05.934782 15619 ts_tablet_manager.cc:1434] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260430 20:02:05.934237 15618 raft_consensus.cc:359] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.935094 15618 raft_consensus.cc:385] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:05.935166 15618 raft_consensus.cc:740] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5034f483700041e397bd6c0ceca4a21c, State: Initialized, Role: FOLLOWER
I20260430 20:02:05.935652 15618 consensus_queue.cc:260] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:05.936435 15599 heartbeater.cc:499] Master 127.10.46.62:46463 was elected leader, sending a full tablet report...
I20260430 20:02:05.937232 15618 ts_tablet_manager.cc:1434] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
W20260430 20:02:06.002242 15330 tablet.cc:2404] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:02:06.048038 15465 tablet.cc:2404] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:02:06.083145 15600 tablet.cc:2404] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:02:06.093489 15623 raft_consensus.cc:493] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:06.093794 15623 raft_consensus.cc:515] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:06.095870 15623 leader_election.cc:290] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8835c3a71d6843edb9d2911616d89e4e (127.10.46.1:39603), 5034f483700041e397bd6c0ceca4a21c (127.10.46.3:39199)
I20260430 20:02:06.100449 15625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.096063 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:39199 (local address 127.10.46.2:57597)
0430 20:02:06.096581 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.096597 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:06.096728 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:06.097041 (+ 313us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:06.097045 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:06.097061 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:06.097316 (+ 255us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:06.097323 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.098883 (+ 1560us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:06.098885 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:06.099939 (+ 1054us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:06.099945 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.100093 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:06.100112 (+ 19us) client_negotiation.cc:770] Sending connection context
0430 20:02:06.100169 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:02:06.100237 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":103,"threads_started":1}
I20260430 20:02:06.100967 15615 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.096582 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:57597 (local address 127.10.46.3:39199)
0430 20:02:06.096724 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.096728 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.096742 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:06.096832 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.096835 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.096922 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.097008 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:06.097615 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.098612 (+ 997us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:06.100129 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.100663 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:06.100704 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.100762 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.100836 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260430 20:02:06.100981 15611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.096058 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:39103 (local address 127.10.46.1:39603)
0430 20:02:06.096386 (+ 328us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.096394 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.096420 (+ 26us) server_negotiation.cc:408] Connection header received
0430 20:02:06.096602 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.096607 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.096674 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.096732 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:06.097615 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.098612 (+ 997us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:06.100129 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.100668 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:06.100704 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.100762 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.100871 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260430 20:02:06.100453 15626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.096004 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39603 (local address 127.10.46.2:39103)
0430 20:02:06.096330 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.096346 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:06.096457 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:06.096886 (+ 429us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:06.096890 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:06.096922 (+ 32us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:06.097260 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:06.097268 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.098878 (+ 1610us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:06.098882 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:06.099902 (+ 1020us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:06.099925 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.100092 (+ 167us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:06.100112 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:02:06.100169 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:02:06.100236 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260430 20:02:06.101827 15553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a10f163aa47e4e0b8ed1850cd186d9a0" candidate_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034f483700041e397bd6c0ceca4a21c" is_pre_election: true
I20260430 20:02:06.102206 15553 raft_consensus.cc:2468] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99efd6a2a7a8423d861ef99d6c30ebfc in term 0.
I20260430 20:02:06.102442 15283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a10f163aa47e4e0b8ed1850cd186d9a0" candidate_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8835c3a71d6843edb9d2911616d89e4e" is_pre_election: true
I20260430 20:02:06.102946 15283 raft_consensus.cc:2468] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99efd6a2a7a8423d861ef99d6c30ebfc in term 0.
I20260430 20:02:06.102912 15353 leader_election.cc:304] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5034f483700041e397bd6c0ceca4a21c, 99efd6a2a7a8423d861ef99d6c30ebfc; no voters:
I20260430 20:02:06.103323 15623 raft_consensus.cc:2804] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:06.103461 15623 raft_consensus.cc:493] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:06.103533 15623 raft_consensus.cc:3060] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:06.105237 15623 raft_consensus.cc:515] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:06.105767 15623 leader_election.cc:290] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [CANDIDATE]: Term 1 election: Requested vote from peers 8835c3a71d6843edb9d2911616d89e4e (127.10.46.1:39603), 5034f483700041e397bd6c0ceca4a21c (127.10.46.3:39199)
I20260430 20:02:06.106302 15283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a10f163aa47e4e0b8ed1850cd186d9a0" candidate_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8835c3a71d6843edb9d2911616d89e4e"
I20260430 20:02:06.106338 15553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a10f163aa47e4e0b8ed1850cd186d9a0" candidate_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034f483700041e397bd6c0ceca4a21c"
I20260430 20:02:06.106431 15283 raft_consensus.cc:3060] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:06.106509 15553 raft_consensus.cc:3060] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:06.107586 15553 raft_consensus.cc:2468] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99efd6a2a7a8423d861ef99d6c30ebfc in term 1.
I20260430 20:02:06.107587 15283 raft_consensus.cc:2468] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99efd6a2a7a8423d861ef99d6c30ebfc in term 1.
I20260430 20:02:06.108119 15351 leader_election.cc:304] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8835c3a71d6843edb9d2911616d89e4e, 99efd6a2a7a8423d861ef99d6c30ebfc; no voters:
I20260430 20:02:06.108350 15623 raft_consensus.cc:2804] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:06.108597 15623 raft_consensus.cc:697] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 1 LEADER]: Becoming Leader. State: Replica: 99efd6a2a7a8423d861ef99d6c30ebfc, State: Running, Role: LEADER
I20260430 20:02:06.109107 15623 consensus_queue.cc:237] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } }
I20260430 20:02:06.113492 15137 catalog_manager.cc:5671] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 99efd6a2a7a8423d861ef99d6c30ebfc (127.10.46.2). New cstate: current_term: 1 leader_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99efd6a2a7a8423d861ef99d6c30ebfc" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 38311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:06.154598 15613 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.150391 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.10.46.2:38311)
0430 20:02:06.150580 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.150583 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.150621 (+ 38us) server_negotiation.cc:408] Connection header received
0430 20:02:06.150716 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.150719 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.150776 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.150862 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:06.151511 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.152168 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:06.153210 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:06.153411 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:06.153506 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:06.154049 (+ 543us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:06.154169 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.154403 (+ 234us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.154461 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260430 20:02:06.162046 15553 raft_consensus.cc:1275] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 1 FOLLOWER]: Refusing update from remote peer 99efd6a2a7a8423d861ef99d6c30ebfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:06.162046 15283 raft_consensus.cc:1275] T a10f163aa47e4e0b8ed1850cd186d9a0 P 8835c3a71d6843edb9d2911616d89e4e [term 1 FOLLOWER]: Refusing update from remote peer 99efd6a2a7a8423d861ef99d6c30ebfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:06.162843 15623 consensus_queue.cc:1048] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5034f483700041e397bd6c0ceca4a21c" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:06.163084 15627 consensus_queue.cc:1048] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8835c3a71d6843edb9d2911616d89e4e" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:06.189781 15136 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:06.190042 15136 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:06.195308 15136 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f92472a2adde440ab221a7815921bca1: Sending DeleteTablet for 3 replicas of tablet a10f163aa47e4e0b8ed1850cd186d9a0
I20260430 20:02:06.196584 15262 tablet_service.cc:1558] Processing DeleteTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:06 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:39644
I20260430 20:02:06.196879 15533 tablet_service.cc:1558] Processing DeleteTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:06 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:44368
I20260430 20:02:06.197103 15398 tablet_service.cc:1558] Processing DeleteTablet for tablet a10f163aa47e4e0b8ed1850cd186d9a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:06 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260430 20:02:06.198226 15641 tablet_replica.cc:333] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: stopping tablet replica
I20260430 20:02:06.198979 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15201
I20260430 20:02:06.201799 15641 raft_consensus.cc:2243] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:06.202289 15641 raft_consensus.cc:2272] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:06.203213 15642 tablet_replica.cc:333] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: stopping tablet replica
I20260430 20:02:06.203564 15642 raft_consensus.cc:2243] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:06.204099 15642 raft_consensus.cc:2272] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:06.204576 15641 ts_tablet_manager.cc:1916] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:06.205945 15642 ts_tablet_manager.cc:1916] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:06.207612 15641 ts_tablet_manager.cc:1929] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:06.207852 15641 log.cc:1199] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/a10f163aa47e4e0b8ed1850cd186d9a0
I20260430 20:02:06.208449 15641 ts_tablet_manager.cc:1950] T a10f163aa47e4e0b8ed1850cd186d9a0 P 5034f483700041e397bd6c0ceca4a21c: Deleting consensus metadata
I20260430 20:02:06.210414 15123 catalog_manager.cc:5002] TS 5034f483700041e397bd6c0ceca4a21c (127.10.46.3:39199): tablet a10f163aa47e4e0b8ed1850cd186d9a0 (table test-table [id=8d2138c93bb94933a6300d01243dfb7d]) successfully deleted
I20260430 20:02:06.210632 15642 ts_tablet_manager.cc:1929] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:06.210812 15642 log.cc:1199] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/a10f163aa47e4e0b8ed1850cd186d9a0
W20260430 20:02:06.210892 15121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.46.1:39603 (error 108)
I20260430 20:02:06.211594 15642 ts_tablet_manager.cc:1950] T a10f163aa47e4e0b8ed1850cd186d9a0 P 99efd6a2a7a8423d861ef99d6c30ebfc: Deleting consensus metadata
I20260430 20:02:06.212565 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15336
I20260430 20:02:06.212841 15614 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.212359 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39603 (local address 127.0.0.1:39660)
0430 20:02:06.212555 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.212684 (+ 129us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260430 20:02:06.213902 15121 catalog_manager.cc:4729] TS 8835c3a71d6843edb9d2911616d89e4e (127.10.46.1:39603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a10f163aa47e4e0b8ed1850cd186d9a0: Network error: Client connection negotiation failed: client connection to 127.10.46.1:39603: connect: Connection refused (error 111)
I20260430 20:02:06.232067 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15471
I20260430 20:02:06.234447 15614 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.232455 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:38311 (local address 127.0.0.1:59772)
0430 20:02:06.234194 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.234282 (+ 88us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.2:38311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1568}
I20260430 20:02:06.241768 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15106
2026-04-30T20:02:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3877 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-04-30T20:02:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:06Z Disabled control of system clock
I20260430 20:02:06.274557 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:45747
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:45747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260430 20:02:06.443631 15647 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:06.444070 15647 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:06.444161 15647 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:06.448693 15647 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:06.448889 15647 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:06.448927 15647 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:06.448975 15647 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:06.449048 15647 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:06.458761 15647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:45747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:06.460122 15647 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:06.461362 15647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:06.467618 15655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:06.467636 15652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:06.467603 15653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:06.468034 15647 server_base.cc:1061] running on GCE node
I20260430 20:02:06.468628 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:06.469693 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:06.470927 15647 hybrid_clock.cc:648] HybridClock initialized: now 1777579326470876 us; error 67 us; skew 500 ppm
I20260430 20:02:06.473263 15647 webserver.cc:492] Webserver started at http://127.10.46.62:33149/ using document root <none> and password file <none>
I20260430 20:02:06.473977 15647 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:06.474069 15647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:06.474301 15647 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:06.476214 15647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "d257674c31ec4389b16d2800bc62a793"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "802e8f388bf7a72a0a64b79b5276cf21"
server_key_iv: "647cb540219c4825b12c4e1af0e01bed"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.476752 15647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "d257674c31ec4389b16d2800bc62a793"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "802e8f388bf7a72a0a64b79b5276cf21"
server_key_iv: "647cb540219c4825b12c4e1af0e01bed"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.480986 15647 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:02:06.484051 15661 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.485231 15647 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260430 20:02:06.485404 15647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "d257674c31ec4389b16d2800bc62a793"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "802e8f388bf7a72a0a64b79b5276cf21"
server_key_iv: "647cb540219c4825b12c4e1af0e01bed"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.485555 15647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:06.490718 15647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:06.491361 15647 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:06.491555 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:06.499514 15647 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:45747
I20260430 20:02:06.499534 15713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:45747 every 8 connection(s)
I20260430 20:02:06.500725 15647 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:06.503558 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15647
I20260430 20:02:06.503707 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:06.504136 15714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:06.504312 10424 external_mini_cluster.cc:1468] Setting key aa04a512a1dd8d00204e9db1785ce50b
I20260430 20:02:06.510965 15717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.505897 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.10.46.62:45747)
0430 20:02:06.506579 (+ 682us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.506591 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.506635 (+ 44us) server_negotiation.cc:408] Connection header received
0430 20:02:06.507775 (+ 1140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.507822 (+ 47us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.508269 (+ 447us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.508654 (+ 385us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:06.509487 (+ 833us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:06.509496 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:06.509512 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:06.509524 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:06.509567 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:06.509583 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:06.509759 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.510049 (+ 290us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.510191 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":119,"threads_started":1}
I20260430 20:02:06.511574 15714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: Bootstrap starting.
I20260430 20:02:06.514703 15714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:06.515461 15714 log.cc:826] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:06.517694 15714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: No bootstrap required, opened a new log
I20260430 20:02:06.520669 15714 raft_consensus.cc:359] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } }
I20260430 20:02:06.520948 15714 raft_consensus.cc:385] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:06.521044 15714 raft_consensus.cc:740] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d257674c31ec4389b16d2800bc62a793, State: Initialized, Role: FOLLOWER
I20260430 20:02:06.521507 15714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } }
I20260430 20:02:06.521637 15714 raft_consensus.cc:399] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:06.521728 15714 raft_consensus.cc:493] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:06.521824 15714 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:06.522878 15714 raft_consensus.cc:515] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } }
I20260430 20:02:06.523232 15714 leader_election.cc:304] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d257674c31ec4389b16d2800bc62a793; no voters:
I20260430 20:02:06.523515 15714 leader_election.cc:290] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:06.523699 15719 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:06.523988 15719 raft_consensus.cc:697] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [term 1 LEADER]: Becoming Leader. State: Replica: d257674c31ec4389b16d2800bc62a793, State: Running, Role: LEADER
I20260430 20:02:06.524317 15719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } }
I20260430 20:02:06.524854 15714 sys_catalog.cc:565] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:06.526108 15721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d257674c31ec4389b16d2800bc62a793. Latest consensus state: current_term: 1 leader_uuid: "d257674c31ec4389b16d2800bc62a793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } } }
I20260430 20:02:06.526103 15720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d257674c31ec4389b16d2800bc62a793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d257674c31ec4389b16d2800bc62a793" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45747 } } }
I20260430 20:02:06.526278 15721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:06.526281 15720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:06.526703 15728 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:06.530316 15728 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:06.537760 15728 catalog_manager.cc:1357] Generated new cluster ID: 25e25f5d6d174c05a9750a99ff880800
I20260430 20:02:06.537869 15728 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:06.552911 15728 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:06.553305 15728 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:06.569370 15728 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: Generated new TSK 0
I20260430 20:02:06.570111 15728 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:02:06.582391 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--builtin_ntp_servers=127.10.46.20:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260430 20:02:06.718713 15738 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:06.718971 15738 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:06.719061 15738 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:06.723687 15738 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:06.723816 15738 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:06.723963 15738 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:06.728520 15738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:06.729816 15738 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:06.731010 15738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:06.739555 15746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:06.739555 15743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:06.739833 15738 server_base.cc:1061] running on GCE node
W20260430 20:02:06.739555 15744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:06.740504 15738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:06.741196 15738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:06.742417 15738 hybrid_clock.cc:648] HybridClock initialized: now 1777579326742403 us; error 49 us; skew 500 ppm
I20260430 20:02:06.744585 15738 webserver.cc:492] Webserver started at http://127.10.46.1:45889/ using document root <none> and password file <none>
I20260430 20:02:06.745239 15738 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:06.745337 15738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:06.745571 15738 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:06.747642 15738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "f4baec30d0a540798e37795bbb46033a"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "33b0e710e2dedada2e1fb295ea4caac6"
server_key_iv: "d53249363a6fecd4200a6ac13c805fbd"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.748363 15738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "f4baec30d0a540798e37795bbb46033a"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "33b0e710e2dedada2e1fb295ea4caac6"
server_key_iv: "d53249363a6fecd4200a6ac13c805fbd"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.752471 15738 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:06.756002 15752 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.757277 15738 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:06.757468 15738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "f4baec30d0a540798e37795bbb46033a"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "33b0e710e2dedada2e1fb295ea4caac6"
server_key_iv: "d53249363a6fecd4200a6ac13c805fbd"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.757725 15738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:06.762960 15738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:06.763602 15738 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:06.763921 15738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:06.764515 15738 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:06.765555 15738 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:06.765683 15738 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.765774 15738 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:06.765825 15738 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.776487 15738 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:39123
I20260430 20:02:06.776506 15865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:39123 every 8 connection(s)
I20260430 20:02:06.777555 15738 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:06.780565 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15738
I20260430 20:02:06.780709 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:06.781040 10424 external_mini_cluster.cc:1468] Setting key 199acd3ac8f4f0f0043598bfc06680ec
I20260430 20:02:06.783720 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--builtin_ntp_servers=127.10.46.20:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260430 20:02:06.783953 15717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.779555 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:34693 (local address 127.10.46.62:45747)
0430 20:02:06.779829 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.779835 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.780889 (+ 1054us) server_negotiation.cc:408] Connection header received
0430 20:02:06.781847 (+ 958us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.781851 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.781889 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.781960 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:06.783201 (+ 1241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:06.783207 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:06.783211 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:06.783232 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:06.783279 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:06.783295 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:06.783474 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.783649 (+ 175us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.783704 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260430 20:02:06.784703 15868 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.779927 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45747 (local address 127.10.46.1:34693)
0430 20:02:06.780715 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.780763 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:06.781643 (+ 880us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:06.782110 (+ 467us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:06.782122 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:06.782201 (+ 79us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:06.782844 (+ 643us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:06.782846 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:06.782871 (+ 25us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:06.782922 (+ 51us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:06.783463 (+ 541us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:06.783469 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:06.783474 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 20:02:06.783657 (+ 183us) client_negotiation.cc:241] Negotiation successful
0430 20:02:06.783761 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260430 20:02:06.786512 15866 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45747
I20260430 20:02:06.786808 15866 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:06.787341 15866 heartbeater.cc:507] Master 127.10.46.62:45747 requested a full tablet report, sending...
I20260430 20:02:06.789165 15678 ts_manager.cc:194] Registered new tserver with Master: f4baec30d0a540798e37795bbb46033a (127.10.46.1:39123)
W20260430 20:02:06.900028 15869 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:06.900285 15869 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:06.900341 15869 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:06.905527 15869 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:06.905639 15869 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:06.905730 15869 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:06.910873 15869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:06.912190 15869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:06.913406 15869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:06.921638 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:06.921653 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:06.921963 15869 server_base.cc:1061] running on GCE node
W20260430 20:02:06.921660 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:06.922505 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:06.923137 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:06.924450 15869 hybrid_clock.cc:648] HybridClock initialized: now 1777579326924420 us; error 39 us; skew 500 ppm
I20260430 20:02:06.926852 15869 webserver.cc:492] Webserver started at http://127.10.46.2:37013/ using document root <none> and password file <none>
I20260430 20:02:06.927598 15869 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:06.927692 15869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:06.928172 15869 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:06.931274 15869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "3f3ae92cba9e4b44b34bf850a2e60498"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "f3320ecaf4ad2c5e1ca17f1dbd6dd631"
server_key_iv: "bb099a97c08462361848f1b19397d87f"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.932435 15869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "3f3ae92cba9e4b44b34bf850a2e60498"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "f3320ecaf4ad2c5e1ca17f1dbd6dd631"
server_key_iv: "bb099a97c08462361848f1b19397d87f"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.936919 15869 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:06.941035 15883 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.942991 15869 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:02:06.943150 15869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "3f3ae92cba9e4b44b34bf850a2e60498"
format_stamp: "Formatted at 2026-04-30 20:02:06 on dist-test-slave-0bdw"
server_key: "f3320ecaf4ad2c5e1ca17f1dbd6dd631"
server_key_iv: "bb099a97c08462361848f1b19397d87f"
server_key_version: "encryptionkey@0"
I20260430 20:02:06.943342 15869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:06.949031 15869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:06.949733 15869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:06.949934 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:06.950770 15869 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:06.951930 15869 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:06.952005 15869 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.952067 15869 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:06.952090 15869 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:06.966004 15869 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:44913
I20260430 20:02:06.966180 15996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:44913 every 8 connection(s)
I20260430 20:02:06.967419 15869 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:06.972688 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 15869
I20260430 20:02:06.973011 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:02:06.973497 10424 external_mini_cluster.cc:1468] Setting key d91824e0de870674368b55379747fc1b
I20260430 20:02:06.974560 15717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.969690 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:54773 (local address 127.10.46.62:45747)
0430 20:02:06.969969 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:06.969972 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:06.971094 (+ 1122us) server_negotiation.cc:408] Connection header received
0430 20:02:06.972037 (+ 943us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:06.972041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:06.972097 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:06.972213 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:06.973604 (+ 1391us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:06.973611 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:06.973614 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:06.973637 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:06.973683 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:06.973695 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:06.974091 (+ 396us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:06.974351 (+ 260us) server_negotiation.cc:300] Negotiation successful
0430 20:02:06.974406 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260430 20:02:06.975957 15999 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:06.970025 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45747 (local address 127.10.46.2:54773)
0430 20:02:06.970820 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:06.970855 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:06.971761 (+ 906us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:06.972411 (+ 650us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:06.972424 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:06.972514 (+ 90us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:06.973283 (+ 769us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:06.973286 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:06.973312 (+ 26us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:06.973364 (+ 52us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:06.974029 (+ 665us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:06.974043 (+ 14us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:06.974050 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:02:06.974311 (+ 261us) client_negotiation.cc:241] Negotiation successful
0430 20:02:06.974592 (+ 281us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":130,"threads_started":1}
I20260430 20:02:06.976807 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--builtin_ntp_servers=127.10.46.20:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260430 20:02:06.977957 15997 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45747
I20260430 20:02:06.978329 15997 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:06.978909 15997 heartbeater.cc:507] Master 127.10.46.62:45747 requested a full tablet report, sending...
I20260430 20:02:06.980540 15678 ts_manager.cc:194] Registered new tserver with Master: 3f3ae92cba9e4b44b34bf850a2e60498 (127.10.46.2:44913)
W20260430 20:02:07.095371 16000 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:07.095628 16000 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:07.095696 16000 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:07.100140 16000 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:07.100268 16000 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:07.100386 16000 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:07.105629 16000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:07.107121 16000 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:07.108440 16000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:07.116954 16005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:07.117667 16006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:07.117609 16008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:07.118048 16000 server_base.cc:1061] running on GCE node
I20260430 20:02:07.118839 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:07.120092 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:07.121294 16000 hybrid_clock.cc:648] HybridClock initialized: now 1777579327121269 us; error 55 us; skew 500 ppm
I20260430 20:02:07.123656 16000 webserver.cc:492] Webserver started at http://127.10.46.3:38055/ using document root <none> and password file <none>
I20260430 20:02:07.124409 16000 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:07.124482 16000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:07.124662 16000 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:07.126580 16000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "70d3a009f3254529bb7a838700d6e654"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "81e82202d56860d5c22d8e4844a65a38"
server_key_iv: "62a8c6333ee20c64fb771754cf83cf8e"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.127240 16000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "70d3a009f3254529bb7a838700d6e654"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "81e82202d56860d5c22d8e4844a65a38"
server_key_iv: "62a8c6333ee20c64fb771754cf83cf8e"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.131136 16000 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:02:07.134451 16014 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:07.136245 16000 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:07.136431 16000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "70d3a009f3254529bb7a838700d6e654"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "81e82202d56860d5c22d8e4844a65a38"
server_key_iv: "62a8c6333ee20c64fb771754cf83cf8e"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.136577 16000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:07.143378 16000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:07.144176 16000 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:07.144409 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:07.145557 16000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:07.146871 16000 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:07.146960 16000 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:07.147047 16000 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:07.147090 16000 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:07.161322 16000 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:41149
I20260430 20:02:07.161355 16127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:41149 every 8 connection(s)
I20260430 20:02:07.164749 16000 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:07.165519 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16000
I20260430 20:02:07.165693 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:07.165994 10424 external_mini_cluster.cc:1468] Setting key abc20828ff424affe807a4626e8c7012
I20260430 20:02:07.172819 15717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.167624 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:56179 (local address 127.10.46.62:45747)
0430 20:02:07.167964 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.167969 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.169226 (+ 1257us) server_negotiation.cc:408] Connection header received
0430 20:02:07.170375 (+ 1149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.170380 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.170452 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.170550 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.172215 (+ 1665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.172219 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.172222 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.172238 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.172265 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.172273 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.172390 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.172588 (+ 198us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.172632 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 20:02:07.173494 16130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.168236 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45747 (local address 127.10.46.3:56179)
0430 20:02:07.169014 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.169057 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.169814 (+ 757us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.170862 (+ 1048us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.170874 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.170951 (+ 77us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.171679 (+ 728us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.171681 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.171709 (+ 28us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.171849 (+ 140us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.172384 (+ 535us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.172392 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.172401 (+ 9us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.172637 (+ 236us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.172767 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
I20260430 20:02:07.175045 16128 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45747
I20260430 20:02:07.175302 16128 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:07.175724 16128 heartbeater.cc:507] Master 127.10.46.62:45747 requested a full tablet report, sending...
I20260430 20:02:07.177970 15678 ts_manager.cc:194] Registered new tserver with Master: 70d3a009f3254529bb7a838700d6e654 (127.10.46.3:41149)
I20260430 20:02:07.183508 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:02:07.189414 15717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.186218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.10.46.62:45747)
0430 20:02:07.186544 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.186548 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.186586 (+ 38us) server_negotiation.cc:408] Connection header received
0430 20:02:07.186971 (+ 385us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.186975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.187020 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.187096 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.188885 (+ 1789us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.188891 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.188894 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.188919 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.188961 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.188970 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.189104 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.189193 (+ 89us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.189238 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260430 20:02:07.194048 15678 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:02:07.197496 15678 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:07.213869 16139 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.210468 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39123 (local address 127.0.0.1:42596)
0430 20:02:07.211078 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.211121 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.211228 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.212033 (+ 805us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.212039 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.212105 (+ 66us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.212750 (+ 645us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.212753 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.212775 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.212835 (+ 60us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.213484 (+ 649us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.213495 (+ 11us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.213502 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.213613 (+ 111us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.213653 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":113,"threads_started":1}
I20260430 20:02:07.214097 16141 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.211227 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:44913 (local address 127.0.0.1:36010)
0430 20:02:07.211587 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.211600 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.211669 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.212493 (+ 824us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.212496 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.212512 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.212970 (+ 458us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.212972 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.212982 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.212999 (+ 17us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.213664 (+ 665us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.213668 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.213672 (+ 4us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.213768 (+ 96us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.213798 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":100,"threads_started":1}
I20260430 20:02:07.214578 16142 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.211240 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:41149 (local address 127.0.0.1:43396)
0430 20:02:07.211858 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.211871 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.211939 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.213108 (+ 1169us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.213111 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.213125 (+ 14us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.213736 (+ 611us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.213740 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.213754 (+ 14us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.213779 (+ 25us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.214360 (+ 581us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.214363 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.214366 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.214434 (+ 68us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.214465 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":125,"threads_started":1}
I20260430 20:02:07.214648 16143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.211643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.10.46.2:44913)
0430 20:02:07.212100 (+ 457us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.212106 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.212125 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:02:07.212182 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.212187 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.212327 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.212497 (+ 170us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.213191 (+ 694us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.213198 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.213212 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.213226 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.213261 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.213276 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.213594 (+ 318us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.213951 (+ 357us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.214492 (+ 541us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":150,"threads_started":1}
I20260430 20:02:07.214928 16140 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.210762 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.10.46.1:39123)
0430 20:02:07.211509 (+ 747us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.211517 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.211543 (+ 26us) server_negotiation.cc:408] Connection header received
0430 20:02:07.211608 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.211619 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.211853 (+ 234us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.212002 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.213289 (+ 1287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.213298 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.213309 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.213323 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.213360 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.213373 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.214290 (+ 917us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.214341 (+ 51us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.214414 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":142,"threads_started":1}
I20260430 20:02:07.215001 16144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.211673 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.10.46.3:41149)
0430 20:02:07.212239 (+ 566us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.212245 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.212266 (+ 21us) server_negotiation.cc:408] Connection header received
0430 20:02:07.212323 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.212328 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.212457 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.212591 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.213914 (+ 1323us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.213920 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.213930 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.214047 (+ 117us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.214088 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.214104 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.214329 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.214777 (+ 448us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.214839 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":153,"threads_started":1}
I20260430 20:02:07.217864 15931 tablet_service.cc:1511] Processing CreateTablet for tablet 09e417c0b5ec415488a8562865c58f7e (DEFAULT_TABLE table=test-table [id=d82c8e7a253e49469697b01a4b2077cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:07.217864 15800 tablet_service.cc:1511] Processing CreateTablet for tablet 09e417c0b5ec415488a8562865c58f7e (DEFAULT_TABLE table=test-table [id=d82c8e7a253e49469697b01a4b2077cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:07.218164 16061 tablet_service.cc:1511] Processing CreateTablet for tablet 09e417c0b5ec415488a8562865c58f7e (DEFAULT_TABLE table=test-table [id=d82c8e7a253e49469697b01a4b2077cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:07.219264 15931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e417c0b5ec415488a8562865c58f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:07.219264 15800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e417c0b5ec415488a8562865c58f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:07.219547 16061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e417c0b5ec415488a8562865c58f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:07.226935 16145 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Bootstrap starting.
I20260430 20:02:07.229038 16147 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Bootstrap starting.
I20260430 20:02:07.229038 16146 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Bootstrap starting.
I20260430 20:02:07.231014 16146 tablet_bootstrap.cc:654] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:07.231091 16145 tablet_bootstrap.cc:654] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:07.231895 16146 log.cc:826] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:07.232307 16147 tablet_bootstrap.cc:654] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:07.232717 16145 log.cc:826] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:07.233541 16147 log.cc:826] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:07.234304 16146 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: No bootstrap required, opened a new log
I20260430 20:02:07.234552 16146 ts_tablet_manager.cc:1403] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 20:02:07.235277 16145 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: No bootstrap required, opened a new log
I20260430 20:02:07.235589 16145 ts_tablet_manager.cc:1403] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260430 20:02:07.236322 16147 tablet_bootstrap.cc:492] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: No bootstrap required, opened a new log
I20260430 20:02:07.236855 16147 ts_tablet_manager.cc:1403] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260430 20:02:07.237572 16146 raft_consensus.cc:359] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.237811 16146 raft_consensus.cc:385] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:07.237869 16146 raft_consensus.cc:740] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4baec30d0a540798e37795bbb46033a, State: Initialized, Role: FOLLOWER
I20260430 20:02:07.238327 16146 consensus_queue.cc:260] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.239074 15866 heartbeater.cc:499] Master 127.10.46.62:45747 was elected leader, sending a full tablet report...
I20260430 20:02:07.239686 16146 ts_tablet_manager.cc:1434] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:02:07.240186 16147 raft_consensus.cc:359] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.240453 16147 raft_consensus.cc:385] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:07.240515 16147 raft_consensus.cc:740] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3ae92cba9e4b44b34bf850a2e60498, State: Initialized, Role: FOLLOWER
I20260430 20:02:07.240657 16145 raft_consensus.cc:359] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.240849 16145 raft_consensus.cc:385] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:07.240908 16145 raft_consensus.cc:740] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d3a009f3254529bb7a838700d6e654, State: Initialized, Role: FOLLOWER
I20260430 20:02:07.241271 16147 consensus_queue.cc:260] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.241269 16145 consensus_queue.cc:260] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.242270 16145 ts_tablet_manager.cc:1434] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260430 20:02:07.242270 16147 ts_tablet_manager.cc:1434] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:02:07.242668 15997 heartbeater.cc:499] Master 127.10.46.62:45747 was elected leader, sending a full tablet report...
I20260430 20:02:07.242662 16128 heartbeater.cc:499] Master 127.10.46.62:45747 was elected leader, sending a full tablet report...
W20260430 20:02:07.278772 15867 tablet.cc:2404] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:02:07.332156 16153 raft_consensus.cc:493] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:07.332353 16153 raft_consensus.cc:515] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.334491 16153 leader_election.cc:290] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4baec30d0a540798e37795bbb46033a (127.10.46.1:39123), 70d3a009f3254529bb7a838700d6e654 (127.10.46.3:41149)
I20260430 20:02:07.340456 15999 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.335003 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39123 (local address 127.10.46.2:41277)
0430 20:02:07.335342 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.335359 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.335455 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.336135 (+ 680us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.336139 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.336154 (+ 15us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.339618 (+ 3464us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.339622 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.339641 (+ 19us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.339688 (+ 47us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.340168 (+ 480us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.340173 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.340180 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.340263 (+ 83us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.340297 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260430 20:02:07.340974 16140 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.335620 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:41277 (local address 127.10.46.1:39123)
0430 20:02:07.335807 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.335813 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.335830 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:07.335880 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.335883 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.335929 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.336018 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.339959 (+ 3941us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.339965 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.339968 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.339983 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.340010 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.340024 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.340123 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.340784 (+ 661us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.340827 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:02:07.341650 15820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09e417c0b5ec415488a8562865c58f7e" candidate_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4baec30d0a540798e37795bbb46033a" is_pre_election: true
I20260430 20:02:07.342087 15820 raft_consensus.cc:2468] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3ae92cba9e4b44b34bf850a2e60498 in term 0.
I20260430 20:02:07.342864 15884 leader_election.cc:304] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3ae92cba9e4b44b34bf850a2e60498, f4baec30d0a540798e37795bbb46033a; no voters:
I20260430 20:02:07.343043 15999 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.340962 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:41149 (local address 127.10.46.2:38961)
0430 20:02:07.341376 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.341393 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.341490 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.341812 (+ 322us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.341815 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.341869 (+ 54us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:07.342499 (+ 630us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:02:07.342503 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:07.342518 (+ 15us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:02:07.342546 (+ 28us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:07.342804 (+ 258us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:07.342807 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:07.342810 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 20:02:07.342893 (+ 83us) client_negotiation.cc:241] Negotiation successful
0430 20:02:07.342925 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300}
I20260430 20:02:07.343044 16144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.340430 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:38961 (local address 127.10.46.3:41149)
0430 20:02:07.341044 (+ 614us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.341048 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.341493 (+ 445us) server_negotiation.cc:408] Connection header received
0430 20:02:07.341591 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.341595 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.341661 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.341754 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.342656 (+ 902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.342660 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.342663 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.342675 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.342697 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.342706 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.342820 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.342890 (+ 70us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.342931 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507}
I20260430 20:02:07.343230 16153 raft_consensus.cc:2804] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:07.343351 16153 raft_consensus.cc:493] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:07.343405 16153 raft_consensus.cc:3060] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:07.344135 16082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09e417c0b5ec415488a8562865c58f7e" candidate_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d3a009f3254529bb7a838700d6e654" is_pre_election: true
I20260430 20:02:07.344998 16082 raft_consensus.cc:2468] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3ae92cba9e4b44b34bf850a2e60498 in term 0.
I20260430 20:02:07.345057 16153 raft_consensus.cc:515] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.345521 16153 leader_election.cc:290] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [CANDIDATE]: Term 1 election: Requested vote from peers f4baec30d0a540798e37795bbb46033a (127.10.46.1:39123), 70d3a009f3254529bb7a838700d6e654 (127.10.46.3:41149)
I20260430 20:02:07.346212 15820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09e417c0b5ec415488a8562865c58f7e" candidate_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4baec30d0a540798e37795bbb46033a"
I20260430 20:02:07.346242 16082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09e417c0b5ec415488a8562865c58f7e" candidate_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d3a009f3254529bb7a838700d6e654"
I20260430 20:02:07.346613 16082 raft_consensus.cc:3060] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:07.346616 15820 raft_consensus.cc:3060] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:07.348179 16082 raft_consensus.cc:2468] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3ae92cba9e4b44b34bf850a2e60498 in term 1.
I20260430 20:02:07.348606 15820 raft_consensus.cc:2468] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3ae92cba9e4b44b34bf850a2e60498 in term 1.
I20260430 20:02:07.348845 15887 leader_election.cc:304] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3ae92cba9e4b44b34bf850a2e60498, 70d3a009f3254529bb7a838700d6e654; no voters:
I20260430 20:02:07.349138 16153 raft_consensus.cc:2804] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:07.349445 16153 raft_consensus.cc:697] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3ae92cba9e4b44b34bf850a2e60498, State: Running, Role: LEADER
I20260430 20:02:07.349915 16153 consensus_queue.cc:237] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } }
I20260430 20:02:07.354036 15678 catalog_manager.cc:5671] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3ae92cba9e4b44b34bf850a2e60498 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3ae92cba9e4b44b34bf850a2e60498" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:07.380232 16143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.378422 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.10.46.2:44913)
0430 20:02:07.378597 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.378603 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.378620 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:07.378708 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.378711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.378769 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.378867 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.379687 (+ 820us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.379691 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.379693 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.379704 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.379725 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.379732 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.379889 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.380027 (+ 138us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.380067 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 20:02:07.387866 15820 raft_consensus.cc:1275] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a [term 1 FOLLOWER]: Refusing update from remote peer 3f3ae92cba9e4b44b34bf850a2e60498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:07.387866 16082 raft_consensus.cc:1275] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 1 FOLLOWER]: Refusing update from remote peer 3f3ae92cba9e4b44b34bf850a2e60498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:07.388908 16153 consensus_queue.cc:1048] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4baec30d0a540798e37795bbb46033a" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:07.389148 16154 consensus_queue.cc:1048] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d3a009f3254529bb7a838700d6e654" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 41149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:07.407737 15678 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:07.407989 15678 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:07.410686 15678 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d257674c31ec4389b16d2800bc62a793: Sending DeleteTablet for 3 replicas of tablet 09e417c0b5ec415488a8562865c58f7e
I20260430 20:02:07.411962 15931 tablet_service.cc:1558] Processing DeleteTablet for tablet 09e417c0b5ec415488a8562865c58f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:07 UTC) from {username='slave'} at 127.0.0.1:36010
I20260430 20:02:07.411998 15800 tablet_service.cc:1558] Processing DeleteTablet for tablet 09e417c0b5ec415488a8562865c58f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:07 UTC) from {username='slave'} at 127.0.0.1:42596
I20260430 20:02:07.412128 16061 tablet_service.cc:1558] Processing DeleteTablet for tablet 09e417c0b5ec415488a8562865c58f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:07 UTC) from {username='slave'} at 127.0.0.1:43396
I20260430 20:02:07.412612 16168 tablet_replica.cc:333] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: stopping tablet replica
I20260430 20:02:07.412855 16169 tablet_replica.cc:333] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: stopping tablet replica
I20260430 20:02:07.412932 16168 raft_consensus.cc:2243] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:07.413110 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15738
I20260430 20:02:07.413355 16168 raft_consensus.cc:2272] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:07.413131 16167 tablet_replica.cc:333] T 09e417c0b5ec415488a8562865c58f7e P f4baec30d0a540798e37795bbb46033a: stopping tablet replica
I20260430 20:02:07.413691 16169 raft_consensus.cc:2243] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:07.414667 16169 raft_consensus.cc:2272] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:07.415268 16168 ts_tablet_manager.cc:1916] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:07.418210 16168 ts_tablet_manager.cc:1929] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:07.418326 16168 log.cc:1199] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/09e417c0b5ec415488a8562865c58f7e
I20260430 20:02:07.418640 16168 ts_tablet_manager.cc:1950] T 09e417c0b5ec415488a8562865c58f7e P 3f3ae92cba9e4b44b34bf850a2e60498: Deleting consensus metadata
I20260430 20:02:07.420202 15662 catalog_manager.cc:5002] TS 3f3ae92cba9e4b44b34bf850a2e60498 (127.10.46.2:44913): tablet 09e417c0b5ec415488a8562865c58f7e (table test-table [id=d82c8e7a253e49469697b01a4b2077cd]) successfully deleted
W20260430 20:02:07.421156 15662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.46.1:39123 (error 108)
I20260430 20:02:07.421486 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15869
I20260430 20:02:07.421622 16169 ts_tablet_manager.cc:1916] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:07.422878 16142 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.422464 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:39123 (local address 127.0.0.1:42612)
0430 20:02:07.422700 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.422765 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260430 20:02:07.423233 15662 catalog_manager.cc:4729] TS f4baec30d0a540798e37795bbb46033a (127.10.46.1:39123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09e417c0b5ec415488a8562865c58f7e: Network error: Client connection negotiation failed: client connection to 127.10.46.1:39123: connect: Connection refused (error 111)
I20260430 20:02:07.425174 16169 ts_tablet_manager.cc:1929] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:07.425313 16169 log.cc:1199] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/09e417c0b5ec415488a8562865c58f7e
I20260430 20:02:07.425786 16169 ts_tablet_manager.cc:1950] T 09e417c0b5ec415488a8562865c58f7e P 70d3a009f3254529bb7a838700d6e654: Deleting consensus metadata
I20260430 20:02:07.427484 15665 catalog_manager.cc:5002] TS 70d3a009f3254529bb7a838700d6e654 (127.10.46.3:41149): tablet 09e417c0b5ec415488a8562865c58f7e (table test-table [id=d82c8e7a253e49469697b01a4b2077cd]) successfully deleted
I20260430 20:02:07.429996 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16000
I20260430 20:02:07.439088 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 15647
2026-04-30T20:02:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1197 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260430 20:02:07.463232 10424 mini_oidc.cc:150] Starting JWKS server
I20260430 20:02:07.464752 10424 webserver.cc:492] Webserver started at https://127.0.0.1:34321/ using document root <none> and password file <none>
I20260430 20:02:07.464887 10424 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 20:02:07.465092 10424 webserver.cc:492] Webserver started at http://127.0.0.1:45041/ using document root <none> and password file <none>
2026-04-30T20:02:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:07Z Disabled control of system clock
I20260430 20:02:07.474786 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37781
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37781
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34321/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/testchainca.cert with env {}
W20260430 20:02:07.589768 16176 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:07.590065 16176 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:07.590190 16176 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:07.594841 16176 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:07.594971 16176 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:07.595005 16176 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:07.595043 16176 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:07.595082 16176 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 20:02:07.595151 16176 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 20:02:07.595191 16176 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:34321/jwks.json
I20260430 20:02:07.603356 16176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34321/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:07.604708 16176 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:07.605564 16176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:07.613075 16184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:07.613214 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:07.613075 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:07.614743 16176 server_base.cc:1061] running on GCE node
I20260430 20:02:07.615398 16176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:07.616704 16176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:07.617944 16176 hybrid_clock.cc:648] HybridClock initialized: now 1777579327617922 us; error 39 us; skew 500 ppm
I20260430 20:02:07.620342 16176 webserver.cc:492] Webserver started at http://127.10.46.62:33471/ using document root <none> and password file <none>
I20260430 20:02:07.621138 16176 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:07.621213 16176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:07.621454 16176 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:07.623354 16176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "aef97426c91540af83a08f0b8de086ed"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "5639a717cd68b4497f5eb9ac129c637a"
server_key_iv: "cdb0a09ae6a4cdbe84ebf55f5a6a8ee0"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.624043 16176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "aef97426c91540af83a08f0b8de086ed"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "5639a717cd68b4497f5eb9ac129c637a"
server_key_iv: "cdb0a09ae6a4cdbe84ebf55f5a6a8ee0"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.627923 16176 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:07.631235 16190 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:07.632956 16176 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:07.633244 16176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "aef97426c91540af83a08f0b8de086ed"
format_stamp: "Formatted at 2026-04-30 20:02:07 on dist-test-slave-0bdw"
server_key: "5639a717cd68b4497f5eb9ac129c637a"
server_key_iv: "cdb0a09ae6a4cdbe84ebf55f5a6a8ee0"
server_key_version: "encryptionkey@0"
I20260430 20:02:07.633388 16176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:07.657760 16176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:07.658632 16176 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:07.658812 16176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:07.670420 16176 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:37781
I20260430 20:02:07.670512 16242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:37781 every 8 connection(s)
I20260430 20:02:07.671734 16176 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:07.672374 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16176
I20260430 20:02:07.672580 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:07.672873 10424 external_mini_cluster.cc:1468] Setting key 7c138d3de7429e635574938638b64950
I20260430 20:02:07.675201 16243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:07.680734 16243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: Bootstrap starting.
I20260430 20:02:07.682471 16246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.674205 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.10.46.62:37781)
0430 20:02:07.674937 (+ 732us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.674947 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.674980 (+ 33us) server_negotiation.cc:408] Connection header received
0430 20:02:07.675691 (+ 711us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.675709 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.676152 (+ 443us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.676497 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:07.677063 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.678223 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:07.678953 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.679262 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:07.680165 (+ 903us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:07.680242 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:07.680261 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:07.681263 (+ 1002us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:07.681308 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:07.681327 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:07.681475 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.681687 (+ 212us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.681865 (+ 178us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":138,"threads_started":1}
I20260430 20:02:07.684265 16243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:07.685459 16243 log.cc:826] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:07.686728 10424 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 20:02:07.689121 16243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: No bootstrap required, opened a new log
I20260430 20:02:07.692190 16243 raft_consensus.cc:359] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } }
I20260430 20:02:07.692446 16243 raft_consensus.cc:385] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:07.692536 16243 raft_consensus.cc:740] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef97426c91540af83a08f0b8de086ed, State: Initialized, Role: FOLLOWER
I20260430 20:02:07.693203 16243 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } }
I20260430 20:02:07.693589 16243 raft_consensus.cc:399] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:07.693728 16243 raft_consensus.cc:493] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:07.693874 16243 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:07.695978 16243 raft_consensus.cc:515] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } }
I20260430 20:02:07.696381 16243 leader_election.cc:304] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef97426c91540af83a08f0b8de086ed; no voters:
I20260430 20:02:07.696722 16243 leader_election.cc:290] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:07.696887 16248 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:07.697162 16248 raft_consensus.cc:697] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [term 1 LEADER]: Becoming Leader. State: Replica: aef97426c91540af83a08f0b8de086ed, State: Running, Role: LEADER
I20260430 20:02:07.697475 16248 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } }
I20260430 20:02:07.697978 16243 sys_catalog.cc:565] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:07.699540 16249 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aef97426c91540af83a08f0b8de086ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } } }
I20260430 20:02:07.699712 16249 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:07.700018 16250 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef97426c91540af83a08f0b8de086ed. Latest consensus state: current_term: 1 leader_uuid: "aef97426c91540af83a08f0b8de086ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef97426c91540af83a08f0b8de086ed" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37781 } } }
I20260430 20:02:07.700111 16250 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed [sys.catalog]: This master's current role is: LEADER
W20260430 20:02:07.704502 16263 catalog_manager.cc:1568] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:07.704650 16263 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:07.704794 16259 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:07.705976 16259 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:07.712494 16259 catalog_manager.cc:1357] Generated new cluster ID: 5883dc9924e040e296795e0df8a50423
I20260430 20:02:07.712592 16259 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:07.738536 16259 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:07.739820 16259 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:07.749068 16259 catalog_manager.cc:6044] T 00000000000000000000000000000000 P aef97426c91540af83a08f0b8de086ed: Generated new TSK 0
I20260430 20:02:07.750027 16259 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:02:07.793486 16246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.765330 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.10.46.62:37781)
0430 20:02:07.765527 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.765533 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.765824 (+ 291us) server_negotiation.cc:408] Connection header received
0430 20:02:07.766297 (+ 473us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.766302 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.766516 (+ 214us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.766643 (+ 127us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:07.767331 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.768134 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:07.769641 (+ 1507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.770323 (+ 682us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:07.770434 (+ 111us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:07.792779 (+ 22345us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0430 20:02:07.792902 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:07.793179 (+ 277us) server_negotiation.cc:300] Negotiation successful
0430 20:02:07.793256 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":160,"threads_started":1}
I20260430 20:02:07.808413 16246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:07.803602 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.10.46.62:37781)
0430 20:02:07.803814 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:07.803891 (+ 77us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:07.803921 (+ 30us) server_negotiation.cc:408] Connection header received
0430 20:02:07.804142 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:07.804145 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:07.804198 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:07.804278 (+ 80us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:07.804887 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.805756 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:07.806784 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.806988 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:07.807040 (+ 52us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:07.808079 (+ 1039us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0430 20:02:07.808291 (+ 212us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44840: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":67}
W20260430 20:02:07.808518 16246 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44840: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260430 20:02:07.809396 16279 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:07.803421 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37781 (local address 127.0.0.1:44840)
0430 20:02:07.803802 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:07.803819 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:07.803944 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:07.804315 (+ 371us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:07.804319 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:07.804329 (+ 10us) client_negotiation.cc:190] Negotiated authn=JWT
0430 20:02:07.804626 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:07.804635 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.805930 (+ 1295us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:07.805935 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:07.806611 (+ 676us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:07.806624 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:07.806766 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:07.806798 (+ 32us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 20:02:07.808312 (+ 1514us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0430 20:02:07.808515 (+ 203us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.46.62:37781: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
I20260430 20:02:22.820770 16285 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:22.815524 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.10.46.62:37781)
0430 20:02:22.815886 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:22.815890 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:22.816111 (+ 221us) server_negotiation.cc:408] Connection header received
0430 20:02:22.816212 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:22.816215 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:22.816282 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:22.816396 (+ 114us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:22.817118 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:22.817723 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:22.818855 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:22.819014 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:22.819058 (+ 44us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:22.820038 (+ 980us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0430 20:02:22.820619 (+ 581us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34474: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
W20260430 20:02:22.820878 16285 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34474: JWT verification failed: token expired
W20260430 20:02:22.821089 16284 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:22.815161 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37781 (local address 127.0.0.1:34474)
0430 20:02:22.815918 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:22.815938 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:22.816111 (+ 173us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:22.816512 (+ 401us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:22.816516 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:22.816527 (+ 11us) client_negotiation.cc:190] Negotiated authn=JWT
0430 20:02:22.816786 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:22.816794 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:22.817863 (+ 1069us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:22.817867 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:22.818687 (+ 820us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:22.818700 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:22.818839 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:22.818871 (+ 32us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 20:02:22.820540 (+ 1669us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0430 20:02:22.820851 (+ 311us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.46.62:37781: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":233,"threads_started":1}
W20260430 20:02:22.824368 16285 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:22.823357 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.10.46.62:37781)
0430 20:02:22.823569 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:22.823576 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:22.823668 (+ 92us) server_negotiation.cc:408] Connection header received
0430 20:02:22.823903 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:22.823906 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:22.823956 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:22.824052 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:22.824256 (+ 204us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34482: BlockingRecv error: recv got EOF from 127.0.0.1:34482 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260430 20:02:22.828476 16295 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.10.46.62:37781
W20260430 20:02:22.829209 16285 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:22.828109 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.10.46.62:37781)
0430 20:02:22.828281 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:22.828287 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:22.828391 (+ 104us) server_negotiation.cc:408] Connection header received
0430 20:02:22.828716 (+ 325us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:22.828720 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:22.828788 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:22.828886 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:22.829064 (+ 178us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34490: BlockingRecv error: recv got EOF from 127.0.0.1:34490 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260430 20:02:22.829480 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16176
2026-04-30T20:02:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15411 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260430 20:02:22.874609 10424 mini_oidc.cc:150] Starting JWKS server
I20260430 20:02:22.875506 10424 webserver.cc:492] Webserver started at https://127.0.0.1:34807/ using document root <none> and password file <none>
I20260430 20:02:22.875583 10424 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 20:02:22.875842 10424 webserver.cc:492] Webserver started at http://127.0.0.1:42805/ using document root <none> and password file <none>
2026-04-30T20:02:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:22Z Disabled control of system clock
I20260430 20:02:22.885063 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:39897
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:39897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34807/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/testchainca.cert with env {}
W20260430 20:02:22.997617 16301 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:22.997880 16301 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:22.997933 16301 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:23.001350 16301 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:23.001422 16301 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:23.001479 16301 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:23.001500 16301 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:23.001518 16301 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 20:02:23.001538 16301 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 20:02:23.001554 16301 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:34807/jwks.json
I20260430 20:02:23.006268 16301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:39897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34807/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:23.007359 16301 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:23.008194 16301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:23.014104 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:23.014125 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:23.014313 16301 server_base.cc:1061] running on GCE node
W20260430 20:02:23.014158 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:23.014978 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:23.016033 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:23.017350 16301 hybrid_clock.cc:648] HybridClock initialized: now 1777579343017315 us; error 37 us; skew 500 ppm
I20260430 20:02:23.020599 16301 webserver.cc:492] Webserver started at http://127.10.46.62:41709/ using document root <none> and password file <none>
I20260430 20:02:23.021267 16301 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:23.021369 16301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:23.021632 16301 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:23.023411 16301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "dd82ed229e084ff2b437ba79415abcc4"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "420c854c531dddca1e0a815ccd8057bc"
server_key_iv: "fcfdeb97ba7f93b0a748daddb2902fa4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.024006 16301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "dd82ed229e084ff2b437ba79415abcc4"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "420c854c531dddca1e0a815ccd8057bc"
server_key_iv: "fcfdeb97ba7f93b0a748daddb2902fa4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.027882 16301 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260430 20:02:23.030720 16315 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:23.031965 16301 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260430 20:02:23.032079 16301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "dd82ed229e084ff2b437ba79415abcc4"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "420c854c531dddca1e0a815ccd8057bc"
server_key_iv: "fcfdeb97ba7f93b0a748daddb2902fa4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.032181 16301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:23.043763 16301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:23.044495 16301 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:23.044654 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:23.052431 16301 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:39897
I20260430 20:02:23.052435 16367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:39897 every 8 connection(s)
I20260430 20:02:23.053690 16301 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:23.057548 16368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:23.062305 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16301
I20260430 20:02:23.062475 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:23.062711 10424 external_mini_cluster.cc:1468] Setting key 6826af667937f7e03420ab76e7aa7d96
I20260430 20:02:23.064392 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4: Bootstrap starting.
I20260430 20:02:23.066803 16368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:23.067566 16368 log.cc:826] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:23.069643 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4: No bootstrap required, opened a new log
I20260430 20:02:23.072338 16368 raft_consensus.cc:359] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } }
I20260430 20:02:23.072571 16368 raft_consensus.cc:385] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:23.072669 16368 raft_consensus.cc:740] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd82ed229e084ff2b437ba79415abcc4, State: Initialized, Role: FOLLOWER
I20260430 20:02:23.073511 16368 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } }
I20260430 20:02:23.073719 16368 raft_consensus.cc:399] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:23.073801 16368 raft_consensus.cc:493] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:23.073904 16368 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:23.075255 16368 raft_consensus.cc:515] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } }
I20260430 20:02:23.075682 16368 leader_election.cc:304] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd82ed229e084ff2b437ba79415abcc4; no voters:
I20260430 20:02:23.076110 16371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.064294 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.10.46.62:39897)
0430 20:02:23.065022 (+ 728us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.065036 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.065085 (+ 49us) server_negotiation.cc:408] Connection header received
0430 20:02:23.065746 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.065764 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.066132 (+ 368us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.066501 (+ 369us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:23.067078 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.067937 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.068645 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.068962 (+ 317us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.073765 (+ 4803us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:23.073801 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:23.073820 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:23.074389 (+ 569us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:23.074442 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:23.074464 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:23.074594 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:23.075060 (+ 466us) server_negotiation.cc:300] Negotiation successful
0430 20:02:23.075245 (+ 185us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":173,"threads_started":1}
I20260430 20:02:23.076123 16368 leader_election.cc:290] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:23.076449 16373 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:23.076666 16373 raft_consensus.cc:697] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [term 1 LEADER]: Becoming Leader. State: Replica: dd82ed229e084ff2b437ba79415abcc4, State: Running, Role: LEADER
I20260430 20:02:23.077244 16373 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } }
I20260430 20:02:23.077628 16368 sys_catalog.cc:565] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:23.078936 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd82ed229e084ff2b437ba79415abcc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } } }
I20260430 20:02:23.079071 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:23.079543 16382 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:23.079859 10424 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 20:02:23.080428 16375 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd82ed229e084ff2b437ba79415abcc4. Latest consensus state: current_term: 1 leader_uuid: "dd82ed229e084ff2b437ba79415abcc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd82ed229e084ff2b437ba79415abcc4" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39897 } } }
I20260430 20:02:23.080549 16375 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:23.083530 16382 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:23.089641 16382 catalog_manager.cc:1357] Generated new cluster ID: 62681045d96247999c2a4ac5171a931e
I20260430 20:02:23.089732 16382 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:23.106199 16382 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:23.107131 16382 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:23.125594 16382 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dd82ed229e084ff2b437ba79415abcc4: Generated new TSK 0
I20260430 20:02:23.126451 16382 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260430 20:02:23.164186 16398 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260430 20:02:23.164285 16398 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260430 20:02:23.164950 16371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.138432 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.10.46.62:39897)
0430 20:02:23.138649 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.138653 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.138823 (+ 170us) server_negotiation.cc:408] Connection header received
0430 20:02:23.138976 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.138979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.139101 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.139242 (+ 141us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:23.139902 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.140490 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.142041 (+ 1551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.142218 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.142316 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:23.164555 (+ 22239us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 20:02:23.164815 (+ 260us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260430 20:02:23.165058 16371 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260430 20:02:23.165148 16397 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:23.138206 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:39897 (local address 127.0.0.1:47012)
0430 20:02:23.138601 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:23.138619 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:23.138779 (+ 160us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:23.139279 (+ 500us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:23.139282 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:23.139292 (+ 10us) client_negotiation.cc:190] Negotiated authn=JWT
0430 20:02:23.139618 (+ 326us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.139628 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.140747 (+ 1119us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.140752 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:23.141774 (+ 1022us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.141801 (+ 27us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.142046 (+ 245us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.142090 (+ 44us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 20:02:23.164793 (+ 22703us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 20:02:23.164976 (+ 183us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.46.62:39897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":186,"threads_started":1}
I20260430 20:02:23.165602 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16301
2026-04-30T20:02:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260430 20:02:23.208082 10424 mini_oidc.cc:150] Starting JWKS server
I20260430 20:02:23.209031 10424 webserver.cc:492] Webserver started at https://127.0.0.1:37015/ using document root <none> and password file <none>
I20260430 20:02:23.209115 10424 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 20:02:23.209290 10424 webserver.cc:492] Webserver started at http://127.0.0.1:44003/ using document root <none> and password file <none>
2026-04-30T20:02:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:23Z Disabled control of system clock
I20260430 20:02:23.220562 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:39385
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:39385
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260430 20:02:23.346645 16404 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:23.346916 16404 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:23.346971 16404 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:23.351388 16404 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:23.351543 16404 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:23.351562 16404 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:23.351581 16404 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:23.351599 16404 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 20:02:23.351619 16404 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 20:02:23.351632 16404 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:37015/jwks.json
I20260430 20:02:23.356549 16404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:39385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:39385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:23.357733 16404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:23.358825 16404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:23.364867 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:23.364897 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:23.365247 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:23.366580 16404 server_base.cc:1061] running on GCE node
I20260430 20:02:23.367326 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:23.369246 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:23.370563 16404 hybrid_clock.cc:648] HybridClock initialized: now 1777579343370541 us; error 51 us; skew 500 ppm
I20260430 20:02:23.373147 16404 webserver.cc:492] Webserver started at http://127.10.46.62:34807/ using document root <none> and password file <none>
I20260430 20:02:23.374068 16404 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:23.374182 16404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:23.374432 16404 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:23.376881 16404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "63078866441947c28f738b728a73d7ca"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "ff648efd8884d1e1f43faeff6fd02a86"
server_key_iv: "45e12b55ea554f6edc906ea32f9d73f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.377460 16404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "63078866441947c28f738b728a73d7ca"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "ff648efd8884d1e1f43faeff6fd02a86"
server_key_iv: "45e12b55ea554f6edc906ea32f9d73f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.381601 16404 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:23.384439 16418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:23.385773 16404 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:23.386055 16404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "63078866441947c28f738b728a73d7ca"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "ff648efd8884d1e1f43faeff6fd02a86"
server_key_iv: "45e12b55ea554f6edc906ea32f9d73f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.386183 16404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:23.406900 16404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:23.407677 16404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:23.407918 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:23.416592 16470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:39385 every 8 connection(s)
I20260430 20:02:23.416568 16404 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:39385
I20260430 20:02:23.417851 16404 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:23.418987 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16404
I20260430 20:02:23.419133 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:23.419410 10424 external_mini_cluster.cc:1468] Setting key d54ea4d7a2aefbcbde1584d545fa00ac
I20260430 20:02:23.422752 16471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:23.430929 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: Bootstrap starting.
I20260430 20:02:23.432649 16474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.421555 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.10.46.62:39385)
0430 20:02:23.422414 (+ 859us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.422428 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.422473 (+ 45us) server_negotiation.cc:408] Connection header received
0430 20:02:23.423475 (+ 1002us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.423503 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.424020 (+ 517us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.424405 (+ 385us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:23.425543 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.427244 (+ 1701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.428554 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.428951 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.429401 (+ 450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:23.429600 (+ 199us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:23.429656 (+ 56us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:23.430385 (+ 729us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:23.430466 (+ 81us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:23.430495 (+ 29us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:23.430690 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:23.431319 (+ 629us) server_negotiation.cc:300] Negotiation successful
0430 20:02:23.431741 (+ 422us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":151,"threads_started":1}
I20260430 20:02:23.434602 16471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:23.436026 16471 log.cc:826] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:23.439179 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: No bootstrap required, opened a new log
I20260430 20:02:23.439615 10424 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 20:02:23.442371 16471 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } }
I20260430 20:02:23.442598 16471 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:23.442673 16471 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63078866441947c28f738b728a73d7ca, State: Initialized, Role: FOLLOWER
I20260430 20:02:23.443186 16471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } }
I20260430 20:02:23.443341 16471 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:23.443428 16471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:23.443836 16471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:23.445057 16471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } }
I20260430 20:02:23.445413 16471 leader_election.cc:304] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63078866441947c28f738b728a73d7ca; no voters:
I20260430 20:02:23.445700 16471 leader_election.cc:290] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:23.445916 16476 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:23.446177 16476 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [term 1 LEADER]: Becoming Leader. State: Replica: 63078866441947c28f738b728a73d7ca, State: Running, Role: LEADER
I20260430 20:02:23.446481 16476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } }
I20260430 20:02:23.447085 16471 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:23.448263 16478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63078866441947c28f738b728a73d7ca. Latest consensus state: current_term: 1 leader_uuid: "63078866441947c28f738b728a73d7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } } }
I20260430 20:02:23.448390 16478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:23.448729 16477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63078866441947c28f738b728a73d7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63078866441947c28f738b728a73d7ca" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 39385 } } }
I20260430 20:02:23.448819 16477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca [sys.catalog]: This master's current role is: LEADER
W20260430 20:02:23.453573 16491 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:23.453671 16491 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:23.453788 16485 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:23.455252 16485 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:23.463373 16485 catalog_manager.cc:1357] Generated new cluster ID: 72c9ea4036d84e5a9f21d5321509b4ba
I20260430 20:02:23.463478 16485 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:23.513113 16485 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:23.514616 16485 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:23.525700 16485 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 63078866441947c28f738b728a73d7ca: Generated new TSK 0
I20260430 20:02:23.526847 16485 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260430 20:02:23.557158 16501 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260430 20:02:23.557247 16501 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260430 20:02:23.559077 16500 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:23.537496 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:39385 (local address 127.0.0.1:33518)
0430 20:02:23.537791 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:23.537803 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:23.537892 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:23.538511 (+ 619us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:23.538514 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:23.538523 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0430 20:02:23.538776 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.538783 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.540042 (+ 1259us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.540049 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:23.540492 (+ 443us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.540500 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.540606 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.540627 (+ 21us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 20:02:23.558774 (+ 18147us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 20:02:23.558934 (+ 160us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.46.62:39385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
I20260430 20:02:23.559064 16474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.537666 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.10.46.62:39385)
0430 20:02:23.537828 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.537833 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.537852 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:02:23.538191 (+ 339us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.538197 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.538313 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.538486 (+ 173us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:23.538971 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.539873 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.540733 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.540991 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.541103 (+ 112us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:23.558506 (+ 17403us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 20:02:23.558906 (+ 400us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260430 20:02:23.559163 16474 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260430 20:02:23.559819 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16404
2026-04-30T20:02:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (393 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-04-30T20:02:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:23Z Disabled control of system clock
I20260430 20:02:23.610591 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35821
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:35821
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/testchainca.cert with env {}
W20260430 20:02:23.734333 16505 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:23.734674 16505 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:23.734772 16505 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:23.738566 16505 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:23.738714 16505 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:23.738749 16505 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:23.738794 16505 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:23.739017 16505 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 20:02:23.739483 16505 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 20:02:23.739554 16505 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260430 20:02:23.746532 16505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:35821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:23.747934 16505 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:23.748881 16505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:23.757405 16510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:23.757405 16511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:23.757676 16513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:23.758196 16505 server_base.cc:1061] running on GCE node
I20260430 20:02:23.760188 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:23.761827 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:23.763087 16505 hybrid_clock.cc:648] HybridClock initialized: now 1777579343763055 us; error 57 us; skew 500 ppm
I20260430 20:02:23.765622 16505 webserver.cc:492] Webserver started at http://127.10.46.62:43867/ using document root <none> and password file <none>
I20260430 20:02:23.766626 16505 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:23.766729 16505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:23.766944 16505 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:23.768895 16505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "aa6473ba152a49f99c6c4c13e72c4b40"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "3c1127092c822ee00f958d40ab1ae901"
server_key_iv: "516cbf3612103215e658b0cf1a80d60d"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.769449 16505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "aa6473ba152a49f99c6c4c13e72c4b40"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "3c1127092c822ee00f958d40ab1ae901"
server_key_iv: "516cbf3612103215e658b0cf1a80d60d"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.773113 16505 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:23.776786 16519 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:23.778203 16505 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:23.778374 16505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "aa6473ba152a49f99c6c4c13e72c4b40"
format_stamp: "Formatted at 2026-04-30 20:02:23 on dist-test-slave-0bdw"
server_key: "3c1127092c822ee00f958d40ab1ae901"
server_key_iv: "516cbf3612103215e658b0cf1a80d60d"
server_key_version: "encryptionkey@0"
I20260430 20:02:23.778501 16505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:23.808108 16505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:23.808856 16505 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:23.809067 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:23.817785 16505 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:35821
I20260430 20:02:23.817963 16571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:35821 every 8 connection(s)
I20260430 20:02:23.818943 16505 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:23.822480 16572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:23.828600 16572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40: Bootstrap starting.
I20260430 20:02:23.829233 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16505
I20260430 20:02:23.829367 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:23.829632 10424 external_mini_cluster.cc:1468] Setting key 163b0d2306a804ca25bfa76a8130c32b
I20260430 20:02:23.831526 16572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:23.832495 16572 log.cc:826] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:23.834559 16572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40: No bootstrap required, opened a new log
I20260430 20:02:23.838435 16572 raft_consensus.cc:359] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } }
I20260430 20:02:23.838866 16572 raft_consensus.cc:385] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:23.838948 16572 raft_consensus.cc:740] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6473ba152a49f99c6c4c13e72c4b40, State: Initialized, Role: FOLLOWER
I20260430 20:02:23.839542 16572 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } }
I20260430 20:02:23.839710 16575 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.830896 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.10.46.62:35821)
0430 20:02:23.831536 (+ 640us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.831688 (+ 152us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.831879 (+ 191us) server_negotiation.cc:408] Connection header received
0430 20:02:23.832778 (+ 899us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.832801 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.833252 (+ 451us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.833716 (+ 464us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:23.834357 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.835248 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.836155 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.836501 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.837684 (+ 1183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:23.837713 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:23.837727 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:02:23.838186 (+ 459us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:23.838225 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:02:23.838242 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:23.838386 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:23.838601 (+ 215us) server_negotiation.cc:300] Negotiation successful
0430 20:02:23.838814 (+ 213us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260430 20:02:23.839752 16572 raft_consensus.cc:399] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:23.839965 16572 raft_consensus.cc:493] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:23.840094 16572 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:23.842190 16572 raft_consensus.cc:515] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } }
I20260430 20:02:23.842649 16572 leader_election.cc:304] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6473ba152a49f99c6c4c13e72c4b40; no voters:
I20260430 20:02:23.843036 16572 leader_election.cc:290] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:23.843142 16577 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:23.843380 16577 raft_consensus.cc:697] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [term 1 LEADER]: Becoming Leader. State: Replica: aa6473ba152a49f99c6c4c13e72c4b40, State: Running, Role: LEADER
I20260430 20:02:23.843734 16577 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } }
I20260430 20:02:23.844250 10424 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 20:02:23.844359 16572 sys_catalog.cc:565] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:23.845504 16578 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } } }
I20260430 20:02:23.845646 16578 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:23.845614 16579 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6473ba152a49f99c6c4c13e72c4b40. Latest consensus state: current_term: 1 leader_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6473ba152a49f99c6c4c13e72c4b40" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35821 } } }
I20260430 20:02:23.845695 16579 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:23.846289 16586 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:23.849470 16586 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:23.855420 16586 catalog_manager.cc:1357] Generated new cluster ID: 92046e4918494832ab6c30bf467f53e5
I20260430 20:02:23.855527 16586 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:23.873987 16586 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:23.874980 16586 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:23.880760 16586 catalog_manager.cc:6044] T 00000000000000000000000000000000 P aa6473ba152a49f99c6c4c13e72c4b40: Generated new TSK 0
I20260430 20:02:23.881392 16586 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:02:23.950572 16575 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:23.896143 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.10.46.62:35821)
0430 20:02:23.896315 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:23.896324 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:23.896518 (+ 194us) server_negotiation.cc:408] Connection header received
0430 20:02:23.896728 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:23.896732 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:23.896794 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:23.896887 (+ 93us) server_negotiation.cc:227] Negotiated authn=JWT
0430 20:02:23.897514 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.898248 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.899429 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.899625 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.899728 (+ 103us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 20:02:23.950182 (+ 50454us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0430 20:02:23.950443 (+ 261us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46906: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260430 20:02:23.950672 16575 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46906: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260430 20:02:23.950824 16601 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:23.896032 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35821 (local address 127.0.0.1:46906)
0430 20:02:23.896382 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:23.896399 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:23.896529 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:23.896928 (+ 399us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:23.896932 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:23.896941 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0430 20:02:23.897260 (+ 319us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.897269 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.898473 (+ 1204us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:23.898479 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:23.899222 (+ 743us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:23.899235 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:23.899389 (+ 154us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:23.899417 (+ 28us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 20:02:23.950434 (+ 51017us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0430 20:02:23.950647 (+ 213us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.46.62:35821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":148,"threads_started":1}
I20260430 20:02:23.951650 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16505
2026-04-30T20:02:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (369 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (150 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:24 dist-test-slave-0bdw krb5kdc[16613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:24 dist-test-slave-0bdw krb5kdc[16613](info): set up 2 sockets
Apr 30 20:02:24 dist-test-slave-0bdw krb5kdc[16613](info): commencing operation
krb5kdc: starting...
W20260430 20:02:26.293004 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:26 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:26.460307 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:45287
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:26.589813 16629 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:26.590108 16629 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:26.590200 16629 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:26.593848 16629 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:26.593956 16629 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:26.594002 16629 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:26.594067 16629 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:26.594115 16629 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:26.598791 16629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:26.600103 16629 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:26.601039 16629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:26.607175 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:26.607196 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:26.607172 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:26.607479 16629 server_base.cc:1061] running on GCE node
I20260430 20:02:26.608525 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:26.609544 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:26.610764 16629 hybrid_clock.cc:648] HybridClock initialized: now 1777579346610742 us; error 48 us; skew 500 ppm
Apr 30 20:02:26 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:26.614545 16629 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:02:26.615936 16629 webserver.cc:492] Webserver started at http://127.10.46.62:43233/ using document root <none> and password file <none>
I20260430 20:02:26.616489 16629 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:26.616557 16629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:26.617041 16629 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:26.619565 16629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "72bfc7079fc143b285fb12ba6bbabc6e"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "2253d65acc62b25c2007cb2b5def5117"
server_key_iv: "eaf415cb262f55ed6ed9202b1fb11eaa"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.620220 16629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "72bfc7079fc143b285fb12ba6bbabc6e"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "2253d65acc62b25c2007cb2b5def5117"
server_key_iv: "eaf415cb262f55ed6ed9202b1fb11eaa"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.624172 16629 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260430 20:02:26.627171 16644 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:26.628737 16629 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:26.628871 16629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "72bfc7079fc143b285fb12ba6bbabc6e"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "2253d65acc62b25c2007cb2b5def5117"
server_key_iv: "eaf415cb262f55ed6ed9202b1fb11eaa"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.629119 16629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:26.641335 16629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:26.644671 16629 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:26.644840 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:26.652189 16629 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:45287
I20260430 20:02:26.652194 16696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:45287 every 8 connection(s)
I20260430 20:02:26.653229 16629 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:26.656224 16697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:26.657568 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16629
I20260430 20:02:26.657732 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:26.657994 10424 external_mini_cluster.cc:1468] Setting key 0879fc70e64898760a2de10177c57b3d
I20260430 20:02:26.663341 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e: Bootstrap starting.
Apr 30 20:02:26 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:26.666421 16697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:26.667291 16697 log.cc:826] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:26.669656 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e: No bootstrap required, opened a new log
I20260430 20:02:26.672159 16700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:26.659459 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.10.46.62:45287)
0430 20:02:26.660004 (+ 545us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:26.660014 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:26.660050 (+ 36us) server_negotiation.cc:408] Connection header received
0430 20:02:26.660633 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:26.660654 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:26.660938 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:26.661261 (+ 323us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:26.662201 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.662954 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:26.663724 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.664049 (+ 325us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:26.666641 (+ 2592us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:26.666658 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:26.666671 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:26.666704 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:26.668943 (+ 2239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:26.669441 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:26.669446 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:26.669452 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:26.669556 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:26.670143 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:26.670146 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:26.670148 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:26.670580 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:26.670772 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:26.671081 (+ 309us) server_negotiation.cc:300] Negotiation successful
0430 20:02:26.671334 (+ 253us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":147,"threads_started":1}
I20260430 20:02:26.673003 16697 raft_consensus.cc:359] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } }
I20260430 20:02:26.673223 16697 raft_consensus.cc:385] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:26.673282 16697 raft_consensus.cc:740] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72bfc7079fc143b285fb12ba6bbabc6e, State: Initialized, Role: FOLLOWER
I20260430 20:02:26.673767 16697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } }
I20260430 20:02:26.673934 16697 raft_consensus.cc:399] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:26.674011 16697 raft_consensus.cc:493] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:26.674189 16697 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:26.676309 16697 raft_consensus.cc:515] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } }
I20260430 20:02:26.676893 16697 leader_election.cc:304] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72bfc7079fc143b285fb12ba6bbabc6e; no voters:
I20260430 20:02:26.677476 16697 leader_election.cc:290] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:26.677657 16702 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:26.677932 16702 raft_consensus.cc:697] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [term 1 LEADER]: Becoming Leader. State: Replica: 72bfc7079fc143b285fb12ba6bbabc6e, State: Running, Role: LEADER
I20260430 20:02:26.678243 16702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } }
I20260430 20:02:26.678783 16697 sys_catalog.cc:565] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:26.680004 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72bfc7079fc143b285fb12ba6bbabc6e. Latest consensus state: current_term: 1 leader_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } } }
I20260430 20:02:26.680153 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:26.680619 16711 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:26.681258 16703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72bfc7079fc143b285fb12ba6bbabc6e" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 45287 } } }
I20260430 20:02:26.681354 16703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:26.684219 16711 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:26.689839 16711 catalog_manager.cc:1357] Generated new cluster ID: d1417996f755445c8adc30dc0f131b24
I20260430 20:02:26.689926 16711 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:26.712918 16711 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:26.713860 16711 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:26.723790 16711 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 72bfc7079fc143b285fb12ba6bbabc6e: Generated new TSK 0
I20260430 20:02:26.724462 16711 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:26.787314 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45287
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:26.893412 16725 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:26.893687 16725 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:26.893780 16725 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:26.897544 16725 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:26.897656 16725 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:26.897775 16725 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:26.902364 16725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:26.903632 16725 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:26.904551 16725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:26.911371 16733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:26.911375 16730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:26.911374 16731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:26.911645 16725 server_base.cc:1061] running on GCE node
I20260430 20:02:26.912088 16725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:26.912662 16725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:26.913842 16725 hybrid_clock.cc:648] HybridClock initialized: now 1777579346913837 us; error 43 us; skew 500 ppm
Apr 30 20:02:26 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:26.917253 16725 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:02:26.918421 16725 webserver.cc:492] Webserver started at http://127.10.46.1:41141/ using document root <none> and password file <none>
I20260430 20:02:26.919018 16725 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:26.919070 16725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:26.919278 16725 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:26.921116 16725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "ba9936d151634c34a0556400a21c5007"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "c479b583c976c8e80ad0a6e1d8623ec2"
server_key_iv: "942a8808323a8038838de1ab60be7e86"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.921636 16725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9936d151634c34a0556400a21c5007"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "c479b583c976c8e80ad0a6e1d8623ec2"
server_key_iv: "942a8808323a8038838de1ab60be7e86"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.925206 16725 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:26.928099 16740 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:26.929786 16725 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260430 20:02:26.929951 16725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "ba9936d151634c34a0556400a21c5007"
format_stamp: "Formatted at 2026-04-30 20:02:26 on dist-test-slave-0bdw"
server_key: "c479b583c976c8e80ad0a6e1d8623ec2"
server_key_iv: "942a8808323a8038838de1ab60be7e86"
server_key_version: "encryptionkey@0"
I20260430 20:02:26.930085 16725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:26.946857 16725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:26.950170 16725 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:26.950430 16725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:26.951054 16725 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:26.952128 16725 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:26.952203 16725 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:26.952278 16725 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:26.952330 16725 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:26.966804 16725 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:40975
I20260430 20:02:26.966818 16853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:40975 every 8 connection(s)
I20260430 20:02:26.967881 16725 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:26.974435 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16725
I20260430 20:02:26.974604 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:26.974893 10424 external_mini_cluster.cc:1468] Setting key ee539fa9e35ce2c220fa8ccbf24814e8
Apr 30 20:02:26 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:26.984244 16700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:26.970487 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:42983 (local address 127.10.46.62:45287)
0430 20:02:26.970649 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:26.970654 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:26.971320 (+ 666us) server_negotiation.cc:408] Connection header received
0430 20:02:26.972316 (+ 996us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:26.972320 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:26.972381 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:26.972500 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:26.974459 (+ 1959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.975560 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:26.976341 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.976531 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:26.979633 (+ 3102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:26.979665 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:26.979671 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:26.979712 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:26.981609 (+ 1897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:26.982423 (+ 814us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:26.982428 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:26.982429 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:26.982479 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:26.982874 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:26.982878 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:26.982879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:26.983047 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:26.983143 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:26.983989 (+ 846us) server_negotiation.cc:300] Negotiation successful
0430 20:02:26.984110 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:02:26.985248 16856 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:26.970704 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45287 (local address 127.10.46.1:42983)
0430 20:02:26.971171 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:26.971207 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:26.972115 (+ 908us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:26.972661 (+ 546us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:26.972676 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:26.973062 (+ 386us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:26.974239 (+ 1177us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:26.974260 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.975727 (+ 1467us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:26.975736 (+ 9us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:26.976220 (+ 484us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:26.976232 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:26.976468 (+ 236us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:26.977120 (+ 652us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:26.977141 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:26.979288 (+ 2147us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:26.981805 (+ 2517us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:26.981816 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:26.981836 (+ 20us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:26.982277 (+ 441us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:26.982588 (+ 311us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:26.982592 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:26.982594 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:26.982739 (+ 145us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:26.983175 (+ 436us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:26.983183 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:26.983532 (+ 349us) client_negotiation.cc:770] Sending connection context
0430 20:02:26.983958 (+ 426us) client_negotiation.cc:241] Negotiation successful
0430 20:02:26.984263 (+ 305us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":107,"threads_started":1}
I20260430 20:02:26.987012 16854 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45287
I20260430 20:02:26.987461 16854 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:26.988165 16854 heartbeater.cc:507] Master 127.10.46.62:45287 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:02:26.990136 16661 ts_manager.cc:194] Registered new tserver with Master: ba9936d151634c34a0556400a21c5007 (127.10.46.1:40975)
I20260430 20:02:26.991343 16661 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:42983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:27.036335 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45287
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:27.145205 16861 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:27.145463 16861 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:27.145524 16861 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:27.149248 16861 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:27.149340 16861 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:27.149470 16861 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:27.154172 16861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:27.155383 16861 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:27.156314 16861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:27.163980 16867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:27.163972 16866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:27.164041 16861 server_base.cc:1061] running on GCE node
W20260430 20:02:27.163972 16869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:27.164636 16861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:27.165238 16861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:27.166514 16861 hybrid_clock.cc:648] HybridClock initialized: now 1777579347166477 us; error 37 us; skew 500 ppm
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.170221 16861 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:02:27.171380 16861 webserver.cc:492] Webserver started at http://127.10.46.2:44655/ using document root <none> and password file <none>
I20260430 20:02:27.172036 16861 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:27.172109 16861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:27.172315 16861 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:27.174045 16861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "f31714ef671545938238bb72876c868e"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "9e3c228bdc19e80deb5f004e1f3acbd3"
server_key_iv: "72719f82c2e108593de1d30c10d9d0a9"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.174608 16861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "f31714ef671545938238bb72876c868e"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "9e3c228bdc19e80deb5f004e1f3acbd3"
server_key_iv: "72719f82c2e108593de1d30c10d9d0a9"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.178535 16861 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 20:02:27.181839 16876 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.182981 16861 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:27.183149 16861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "f31714ef671545938238bb72876c868e"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "9e3c228bdc19e80deb5f004e1f3acbd3"
server_key_iv: "72719f82c2e108593de1d30c10d9d0a9"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.183283 16861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:27.208774 16861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:27.212635 16861 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:27.212872 16861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:27.213474 16861 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:27.214596 16861 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:27.214802 16861 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.215013 16861 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:27.215075 16861 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.226370 16861 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:42599
I20260430 20:02:27.226508 16989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:42599 every 8 connection(s)
I20260430 20:02:27.227437 16861 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:27.233464 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16861
I20260430 20:02:27.233601 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:02:27.233893 10424 external_mini_cluster.cc:1468] Setting key b41608a1f633c227c1752a643510e1f9
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:27.242403 16700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.229399 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:60613 (local address 127.10.46.62:45287)
0430 20:02:27.229546 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.229550 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.230595 (+ 1045us) server_negotiation.cc:408] Connection header received
0430 20:02:27.231851 (+ 1256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.231855 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.231931 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.232041 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.233962 (+ 1921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.234564 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.235546 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.235891 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.238217 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:27.238243 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:27.238246 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:27.238271 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:27.240366 (+ 2095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.240943 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.240948 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.240950 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.241003 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.241315 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.241318 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.241320 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.241486 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:27.241598 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:27.242108 (+ 510us) server_negotiation.cc:300] Negotiation successful
0430 20:02:27.242234 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 20:02:27.243039 16992 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.229665 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45287 (local address 127.10.46.2:60613)
0430 20:02:27.230338 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:27.230397 (+ 59us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:27.231447 (+ 1050us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:27.232205 (+ 758us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:27.232213 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:27.232629 (+ 416us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:27.233720 (+ 1091us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:27.233740 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.234725 (+ 985us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.234729 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:27.235303 (+ 574us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:27.235312 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.235517 (+ 205us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.236151 (+ 634us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:27.236170 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:27.238044 (+ 1874us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:27.240550 (+ 2506us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:27.240557 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:27.240567 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:27.240826 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:27.241109 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:27.241113 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:27.241114 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:27.241206 (+ 92us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:27.241611 (+ 405us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:27.241619 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:27.241864 (+ 245us) client_negotiation.cc:770] Sending connection context
0430 20:02:27.242075 (+ 211us) client_negotiation.cc:241] Negotiation successful
0430 20:02:27.242314 (+ 239us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:02:27.244235 16990 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45287
I20260430 20:02:27.244524 16990 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:27.245042 16990 heartbeater.cc:507] Master 127.10.46.62:45287 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
I20260430 20:02:27.246438 16661 ts_manager.cc:194] Registered new tserver with Master: f31714ef671545938238bb72876c868e (127.10.46.2:42599)
I20260430 20:02:27.247040 16661 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:60613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:27.292270 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:45287
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:27.400152 16997 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:27.400517 16997 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:27.400599 16997 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:27.404165 16997 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:27.404239 16997 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:27.404358 16997 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:27.408839 16997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:27.410006 16997 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:27.410868 16997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:27.417989 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:27.418021 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:27.417990 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:27.418489 16997 server_base.cc:1061] running on GCE node
I20260430 20:02:27.418907 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:27.419480 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:27.420640 16997 hybrid_clock.cc:648] HybridClock initialized: now 1777579347420482 us; error 174 us; skew 500 ppm
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.423491 16997 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:02:27.424625 16997 webserver.cc:492] Webserver started at http://127.10.46.3:40563/ using document root <none> and password file <none>
I20260430 20:02:27.425187 16997 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:27.425261 16997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:27.425489 16997 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:27.427183 16997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "2b84eb00edc649529f4eb02ff552171f"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "50b397703b4ee5f0768c310f804a2452"
server_key_iv: "f1378ac7cce5ebc8695ba5b90129d7b8"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.427685 16997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "2b84eb00edc649529f4eb02ff552171f"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "50b397703b4ee5f0768c310f804a2452"
server_key_iv: "f1378ac7cce5ebc8695ba5b90129d7b8"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.431102 16997 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:27.433581 17012 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.434774 16997 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 20:02:27.434919 16997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "2b84eb00edc649529f4eb02ff552171f"
format_stamp: "Formatted at 2026-04-30 20:02:27 on dist-test-slave-0bdw"
server_key: "50b397703b4ee5f0768c310f804a2452"
server_key_iv: "f1378ac7cce5ebc8695ba5b90129d7b8"
server_key_version: "encryptionkey@0"
I20260430 20:02:27.435035 16997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:27.464124 16997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:27.467116 16997 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:27.467343 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:27.468017 16997 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:27.468955 16997 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:27.469029 16997 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.469122 16997 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:27.469159 16997 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:27.480178 16997 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:41345
I20260430 20:02:27.480207 17125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:41345 every 8 connection(s)
I20260430 20:02:27.481235 16997 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:27.489928 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 16997
I20260430 20:02:27.490191 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:27.490523 10424 external_mini_cluster.cc:1468] Setting key 7a99bd5a1164cfda5ca61b25aa600e78
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:27.495024 16700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.483075 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:34565 (local address 127.10.46.62:45287)
0430 20:02:27.483219 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.483223 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.484143 (+ 920us) server_negotiation.cc:408] Connection header received
0430 20:02:27.485034 (+ 891us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.485038 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.485102 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.485205 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.486861 (+ 1656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.487356 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.488066 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.488238 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.491483 (+ 3245us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:27.491503 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:27.491508 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:27.491568 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:27.493112 (+ 1544us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.493620 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.493624 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.493628 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.493674 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.493975 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.493979 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.493981 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.494139 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:27.494242 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:27.494737 (+ 495us) server_negotiation.cc:300] Negotiation successful
0430 20:02:27.494858 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 20:02:27.495637 17128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.483469 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:45287 (local address 127.10.46.3:34565)
0430 20:02:27.484007 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:27.484039 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:27.484833 (+ 794us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:27.485371 (+ 538us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:27.485380 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:27.485865 (+ 485us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:27.486702 (+ 837us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:27.486723 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.487485 (+ 762us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.487489 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:27.487950 (+ 461us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:27.487958 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.488175 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.488851 (+ 676us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:27.488877 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:27.491322 (+ 2445us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:27.493245 (+ 1923us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:27.493252 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:27.493264 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:27.493513 (+ 249us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:27.493778 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:27.493781 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:27.493783 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:27.493882 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:27.494255 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:27.494262 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:27.494493 (+ 231us) client_negotiation.cc:770] Sending connection context
0430 20:02:27.494697 (+ 204us) client_negotiation.cc:241] Negotiation successful
0430 20:02:27.494932 (+ 235us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":108,"threads_started":1}
I20260430 20:02:27.496659 17126 heartbeater.cc:344] Connected to a master server at 127.10.46.62:45287
I20260430 20:02:27.496901 17126 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:27.497397 17126 heartbeater.cc:507] Master 127.10.46.62:45287 requested a full tablet report, sending...
I20260430 20:02:27.498409 16661 ts_manager.cc:194] Registered new tserver with Master: 2b84eb00edc649529f4eb02ff552171f (127.10.46.3:41345)
I20260430 20:02:27.498976 16661 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:34565
I20260430 20:02:27.504673 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/test-admin.keytab.
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.521189 10424 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260430 20:02:27.524017 17133 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:27.522730 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.10.46.1:40975)
0430 20:02:27.523015 (+ 285us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.523021 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.523044 (+ 23us) server_negotiation.cc:408] Connection header received
0430 20:02:27.523099 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.523103 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.523260 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.523380 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.523882 (+ 502us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40748: BlockingRecv error: recv got EOF from 127.0.0.1:40748 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
W20260430 20:02:27.524266 10424 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc' of type FILE
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.525398 10424 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.526379 10424 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:02:27.537420 17133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.527353 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.10.46.1:40975)
0430 20:02:27.527528 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.527532 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.527618 (+ 86us) server_negotiation.cc:408] Connection header received
0430 20:02:27.527722 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.527725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.527820 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.527903 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.528594 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.529330 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.530009 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.530260 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.533120 (+ 2860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:27.533167 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:27.533183 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:27.533228 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:27.535355 (+ 2127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.535941 (+ 586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.535946 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.535948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.536000 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.536520 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.536522 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.536524 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.536711 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:27.536879 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:27.537125 (+ 246us) server_negotiation.cc:300] Negotiation successful
0430 20:02:27.537248 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260430 20:02:27.541445 17133 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:27.540376 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.10.46.1:40975)
0430 20:02:27.540511 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.540514 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.540634 (+ 120us) server_negotiation.cc:408] Connection header received
0430 20:02:27.540769 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.540772 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.540818 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.540897 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.541301 (+ 404us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40758: BlockingRecv error: recv got EOF from 127.0.0.1:40758 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260430 20:02:27.541548 10424 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc' of type FILE
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.542883 10424 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.543936 10424 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:02:27.553201 17133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.544931 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.10.46.1:40975)
0430 20:02:27.545041 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.545045 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.545172 (+ 127us) server_negotiation.cc:408] Connection header received
0430 20:02:27.545215 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.545217 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.545262 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.545335 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.546120 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.546639 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.547568 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.547743 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.550064 (+ 2321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:27.550096 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:27.550098 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:27.550125 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:27.551560 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.552081 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.552084 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.552086 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.552130 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.552387 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.552392 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.552398 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.552533 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:27.552684 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:27.552923 (+ 239us) server_negotiation.cc:300] Negotiation successful
0430 20:02:27.553028 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260430 20:02:27.556727 17133 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:27.555429 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.10.46.1:40975)
0430 20:02:27.555584 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.555587 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.555599 (+ 12us) server_negotiation.cc:408] Connection header received
0430 20:02:27.555712 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.555714 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.555806 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.555884 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.556603 (+ 719us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40778: BlockingRecv error: recv got EOF from 127.0.0.1:40778 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260430 20:02:27.557106 10424 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777579284342945-10424-0/krb5kdc/krb5cc' of type FILE
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.558377 10424 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:27.559433 10424 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[16613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.1@KRBTEST.COM
I20260430 20:02:27.568702 17133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:27.560635 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.10.46.1:40975)
0430 20:02:27.560790 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:27.560794 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:27.560808 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:27.560908 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:27.560910 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:27.560952 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:27.561023 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:27.561687 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.562162 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:27.562966 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:27.563109 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:27.565296 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:27.565317 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:27.565320 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:27.565357 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:27.567109 (+ 1752us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.567571 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.567575 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.567577 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.567628 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:27.567915 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:27.567919 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:27.567922 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:27.568064 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:27.568198 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:27.568420 (+ 222us) server_negotiation.cc:300] Negotiation successful
0430 20:02:27.568534 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:02:27.570145 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16725
I20260430 20:02:27.576170 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16861
I20260430 20:02:27.582540 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16997
I20260430 20:02:27.589051 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 16629
2026-04-30T20:02:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3359 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[17146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[17146](info): set up 2 sockets
Apr 30 20:02:27 dist-test-slave-0bdw krb5kdc[17146](info): commencing operation
krb5kdc: starting...
W20260430 20:02:29.641191 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:29 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:29.800602 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37669
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:29.908244 17162 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:29.908535 17162 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:29.908628 17162 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:29.912077 17162 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:29.912181 17162 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:29.912225 17162 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:29.912287 17162 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:29.912348 17162 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:29.917047 17162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:29.918188 17162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:29.919085 17162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:29.924917 17167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:29.924918 17168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:29.924918 17170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:29.925312 17162 server_base.cc:1061] running on GCE node
I20260430 20:02:29.925943 17162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:29.926862 17162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:29.928059 17162 hybrid_clock.cc:648] HybridClock initialized: now 1777579349928028 us; error 46 us; skew 500 ppm
Apr 30 20:02:29 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:29.931209 17162 init.cc:377] Logged in from keytab as oryx/127.10.46.62@KRBTEST.COM (short username oryx)
I20260430 20:02:29.932636 17162 webserver.cc:492] Webserver started at http://127.10.46.62:42335/ using document root <none> and password file <none>
I20260430 20:02:29.933215 17162 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:29.933267 17162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:29.933496 17162 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:29.935216 17162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "0fd426cc2dce4b249bbd545a803b624b"
format_stamp: "Formatted at 2026-04-30 20:02:29 on dist-test-slave-0bdw"
server_key: "353adc1268304ec85b8c888c5517cb07"
server_key_iv: "7d5dda1f1ebe7c81a39cb9a8dd7269dc"
server_key_version: "encryptionkey@0"
I20260430 20:02:29.935750 17162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "0fd426cc2dce4b249bbd545a803b624b"
format_stamp: "Formatted at 2026-04-30 20:02:29 on dist-test-slave-0bdw"
server_key: "353adc1268304ec85b8c888c5517cb07"
server_key_iv: "7d5dda1f1ebe7c81a39cb9a8dd7269dc"
server_key_version: "encryptionkey@0"
I20260430 20:02:29.940124 17162 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:29.942582 17177 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:29.943928 17162 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:02:29.944085 17162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "0fd426cc2dce4b249bbd545a803b624b"
format_stamp: "Formatted at 2026-04-30 20:02:29 on dist-test-slave-0bdw"
server_key: "353adc1268304ec85b8c888c5517cb07"
server_key_iv: "7d5dda1f1ebe7c81a39cb9a8dd7269dc"
server_key_version: "encryptionkey@0"
I20260430 20:02:29.944208 17162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:29.964486 17162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:29.971697 17162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:29.971974 17162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:29.980592 17162 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:37669
I20260430 20:02:29.980589 17229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:37669 every 8 connection(s)
I20260430 20:02:29.981715 17162 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:29.984669 17230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:29.987147 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17162
I20260430 20:02:29.987260 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:29.987533 10424 external_mini_cluster.cc:1468] Setting key 1f10f638421a64e271a6a2a67f3de12d
I20260430 20:02:29.991006 17230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Bootstrap starting.
I20260430 20:02:29.993270 17230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:29.994016 17230 log.cc:826] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:29.996343 17230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: No bootstrap required, opened a new log
Apr 30 20:02:29 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:29.999938 17230 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } }
I20260430 20:02:30.000195 17230 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:30.000295 17230 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd426cc2dce4b249bbd545a803b624b, State: Initialized, Role: FOLLOWER
I20260430 20:02:30.000720 17230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } }
I20260430 20:02:30.000835 17230 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:30.000921 17230 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:30.001012 17230 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:30.002460 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:29.989222 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.10.46.62:37669)
0430 20:02:29.989775 (+ 553us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:29.989785 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:29.989821 (+ 36us) server_negotiation.cc:408] Connection header received
0430 20:02:29.990340 (+ 519us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:29.990358 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:29.990646 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:29.991007 (+ 361us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:29.991966 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:29.993086 (+ 1120us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:29.994200 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:29.994530 (+ 330us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:29.997233 (+ 2703us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:29.997256 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:29.997271 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:29.997305 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:29.999395 (+ 2090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:29.999954 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:29.999963 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:29.999971 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.000068 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.000495 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.000498 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.000499 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.000943 (+ 444us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:30.001244 (+ 301us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:30.001621 (+ 377us) server_negotiation.cc:300] Negotiation successful
0430 20:02:30.001855 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":186,"threads_started":1}
I20260430 20:02:30.002440 17230 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } }
I20260430 20:02:30.002838 17230 leader_election.cc:304] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd426cc2dce4b249bbd545a803b624b; no voters:
I20260430 20:02:30.003149 17230 leader_election.cc:290] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:30.003296 17235 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:30.003522 17235 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 LEADER]: Becoming Leader. State: Replica: 0fd426cc2dce4b249bbd545a803b624b, State: Running, Role: LEADER
I20260430 20:02:30.003858 17235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } }
I20260430 20:02:30.004475 17230 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:30.005074 17237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd426cc2dce4b249bbd545a803b624b. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } }
I20260430 20:02:30.005203 17237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:30.005391 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } }
I20260430 20:02:30.005506 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:30.005625 17240 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:30.008430 17240 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:30.014006 17240 catalog_manager.cc:1357] Generated new cluster ID: 378f4f0d68054f2184a1aeb75cd4f491
I20260430 20:02:30.014092 17240 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:30.062525 17240 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:30.063809 17240 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:30.074798 17240 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Generated new TSK 0
I20260430 20:02:30.075523 17240 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:30.136291 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:37669
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:30.256598 17258 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:30.256855 17258 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:30.256945 17258 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:30.260483 17258 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:30.260560 17258 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:30.260643 17258 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:30.265292 17258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:30.266492 17258 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:30.267349 17258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:30.274151 17266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:30.274151 17263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:30.274245 17258 server_base.cc:1061] running on GCE node
W20260430 20:02:30.274151 17264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:30.274919 17258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:30.275501 17258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:30.276726 17258 hybrid_clock.cc:648] HybridClock initialized: now 1777579350276670 us; error 72 us; skew 500 ppm
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:30.280351 17258 init.cc:377] Logged in from keytab as oryx/127.10.46.1@KRBTEST.COM (short username oryx)
I20260430 20:02:30.281519 17258 webserver.cc:492] Webserver started at http://127.10.46.1:35689/ using document root <none> and password file <none>
I20260430 20:02:30.282124 17258 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:30.282205 17258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:30.282426 17258 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:30.284255 17258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "29adfe52affa45fc84188f9ae739e46f"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "8e77950bbb876a578947ea5d56ac645e"
server_key_iv: "68ef09507e36039fed23d89c3a0843cb"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.284786 17258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "29adfe52affa45fc84188f9ae739e46f"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "8e77950bbb876a578947ea5d56ac645e"
server_key_iv: "68ef09507e36039fed23d89c3a0843cb"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.288513 17258 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:30.291191 17273 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.292348 17258 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:02:30.292485 17258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "29adfe52affa45fc84188f9ae739e46f"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "8e77950bbb876a578947ea5d56ac645e"
server_key_iv: "68ef09507e36039fed23d89c3a0843cb"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.292606 17258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:30.317576 17258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:30.321717 17258 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:30.321949 17258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:30.322571 17258 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:30.323501 17258 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:30.323550 17258 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.323589 17258 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:30.323606 17258 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.333962 17258 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:45363
I20260430 20:02:30.333981 17386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:45363 every 8 connection(s)
I20260430 20:02:30.335074 17258 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:30.344487 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17258
I20260430 20:02:30.344631 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:30.344902 10424 external_mini_cluster.cc:1468] Setting key a45dbf2191ad407da36dc0777c864e74
I20260430 20:02:30.348382 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.336970 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:49721 (local address 127.10.46.62:37669)
0430 20:02:30.337167 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:30.337170 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:30.337756 (+ 586us) server_negotiation.cc:408] Connection header received
0430 20:02:30.338616 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:30.338620 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:30.338714 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:30.338809 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:30.340125 (+ 1316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.340650 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.341296 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.341473 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.344345 (+ 2872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:30.344368 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:30.344370 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:30.344399 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:30.346096 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.346730 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.346734 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.346736 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.346798 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.347150 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.347152 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.347153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.347355 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:30.347444 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:30.348107 (+ 663us) server_negotiation.cc:300] Negotiation successful
0430 20:02:30.348221 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260430 20:02:30.349293 17389 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.337190 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:49721)
0430 20:02:30.337608 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:30.337643 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:30.338409 (+ 766us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:30.338960 (+ 551us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:30.338967 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:30.339358 (+ 391us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:30.339964 (+ 606us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.339978 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.340782 (+ 804us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.340792 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:30.341175 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.341181 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.341410 (+ 229us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.342541 (+ 1131us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:30.342560 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:30.344161 (+ 1601us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:30.346236 (+ 2075us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.346246 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.346262 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.346567 (+ 305us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.346907 (+ 340us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.346911 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.346913 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.347045 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.347481 (+ 436us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:30.347490 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:30.347819 (+ 329us) client_negotiation.cc:770] Sending connection context
0430 20:02:30.348062 (+ 243us) client_negotiation.cc:241] Negotiation successful
0430 20:02:30.348331 (+ 269us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260430 20:02:30.350508 17387 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:30.350801 17387 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:30.351346 17387 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:30.352903 17190 ts_manager.cc:194] Registered new tserver with Master: 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363)
I20260430 20:02:30.354063 17190 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:49721
WARNING: no policy specified for oryx/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:30.407974 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:37669
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:30.517612 17394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:30.517915 17394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:30.518008 17394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:30.522269 17394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:30.522403 17394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:30.522528 17394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:30.527266 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:30.528523 17394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:30.529371 17394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:30.535920 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:30.535933 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:30.536192 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:30.536350 17394 server_base.cc:1061] running on GCE node
I20260430 20:02:30.536783 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:30.537412 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:30.538662 17394 hybrid_clock.cc:648] HybridClock initialized: now 1777579350538637 us; error 44 us; skew 500 ppm
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:30.541710 17394 init.cc:377] Logged in from keytab as oryx/127.10.46.2@KRBTEST.COM (short username oryx)
I20260430 20:02:30.542837 17394 webserver.cc:492] Webserver started at http://127.10.46.2:32823/ using document root <none> and password file <none>
I20260430 20:02:30.543426 17394 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:30.543514 17394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:30.543744 17394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:30.545486 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "a1092265a9534ad5b673c04b278c87c3"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "0527a32cd1e80ec6bc134334ac18e77e"
server_key_iv: "579c42d9b791652cdccb1f4ac9f3ad98"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.546003 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "a1092265a9534ad5b673c04b278c87c3"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "0527a32cd1e80ec6bc134334ac18e77e"
server_key_iv: "579c42d9b791652cdccb1f4ac9f3ad98"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.549439 17394 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260430 20:02:30.551667 17409 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.552703 17394 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 20:02:30.552857 17394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "a1092265a9534ad5b673c04b278c87c3"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "0527a32cd1e80ec6bc134334ac18e77e"
server_key_iv: "579c42d9b791652cdccb1f4ac9f3ad98"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.552968 17394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:30.568902 17394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:30.571764 17394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:30.572074 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:30.572691 17394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:30.573603 17394 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:30.573688 17394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.573758 17394 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:30.573801 17394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.584201 17394 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:43023
I20260430 20:02:30.584216 17522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:43023 every 8 connection(s)
I20260430 20:02:30.585295 17394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:30.594895 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17394
I20260430 20:02:30.595029 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:30.595314 10424 external_mini_cluster.cc:1468] Setting key 2f0d8906fbc224ec9639691e8632cd54
I20260430 20:02:30.601085 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.587280 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:49295 (local address 127.10.46.62:37669)
0430 20:02:30.587456 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:30.587460 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:30.588220 (+ 760us) server_negotiation.cc:408] Connection header received
0430 20:02:30.589069 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:30.589073 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:30.589125 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:30.589313 (+ 188us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:30.590564 (+ 1251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.591078 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.591883 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.592038 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.596050 (+ 4012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:30.596067 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:30.596069 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:30.596107 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:30.598400 (+ 2293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.599159 (+ 759us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.599167 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.599171 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.599247 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.599559 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.599565 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.599570 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.599819 (+ 249us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:30.599935 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:30.600664 (+ 729us) server_negotiation.cc:300] Negotiation successful
0430 20:02:30.600837 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 20:02:30.601567 17525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.587569 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:49295)
0430 20:02:30.588083 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:30.588116 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:30.588872 (+ 756us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:30.589484 (+ 612us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:30.589493 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:30.589896 (+ 403us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:30.590360 (+ 464us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.590372 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.591230 (+ 858us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.591234 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:30.591652 (+ 418us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.591659 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.591914 (+ 255us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.593298 (+ 1384us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:30.593329 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:30.595885 (+ 2556us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:30.598710 (+ 2825us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.598717 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.598727 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.599040 (+ 313us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.599349 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.599351 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.599353 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.599466 (+ 113us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.599947 (+ 481us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:30.599955 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:30.600273 (+ 318us) client_negotiation.cc:770] Sending connection context
0430 20:02:30.600566 (+ 293us) client_negotiation.cc:241] Negotiation successful
0430 20:02:30.600814 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":149,"threads_started":1}
I20260430 20:02:30.602797 17523 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:30.603063 17523 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:30.603585 17523 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:30.604794 17190 ts_manager.cc:194] Registered new tserver with Master: a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
I20260430 20:02:30.605607 17190 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:49295
WARNING: no policy specified for oryx/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:30.656430 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:37669
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:30.762745 17530 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:30.762992 17530 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:30.763051 17530 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:30.766494 17530 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:30.766570 17530 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:30.766692 17530 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:30.771332 17530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:30.772594 17530 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:30.773450 17530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:30.779928 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:30.779924 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:30.779928 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:30.780272 17530 server_base.cc:1061] running on GCE node
I20260430 20:02:30.780714 17530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:30.781224 17530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:30.782405 17530 hybrid_clock.cc:648] HybridClock initialized: now 1777579350782381 us; error 47 us; skew 500 ppm
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:30.785245 17530 init.cc:377] Logged in from keytab as oryx/127.10.46.3@KRBTEST.COM (short username oryx)
I20260430 20:02:30.786324 17530 webserver.cc:492] Webserver started at http://127.10.46.3:39809/ using document root <none> and password file <none>
I20260430 20:02:30.786849 17530 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:30.786895 17530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:30.787046 17530 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:30.789193 17530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "63ee5bda423347388be745e1042e726d"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "35988facc1e4fcfb811ab669d9da1814"
server_key_iv: "60cf062a750a2acdb9ccd2d51a5cdb0f"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.789726 17530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "63ee5bda423347388be745e1042e726d"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "35988facc1e4fcfb811ab669d9da1814"
server_key_iv: "60cf062a750a2acdb9ccd2d51a5cdb0f"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.793141 17530 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:30.795307 17545 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.796515 17530 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:02:30.796667 17530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "63ee5bda423347388be745e1042e726d"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "35988facc1e4fcfb811ab669d9da1814"
server_key_iv: "60cf062a750a2acdb9ccd2d51a5cdb0f"
server_key_version: "encryptionkey@0"
I20260430 20:02:30.796775 17530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:30.815639 17530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:30.819222 17530 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:30.819448 17530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:30.820116 17530 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:30.821048 17530 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:30.821099 17530 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.821141 17530 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:30.821199 17530 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:30.831482 17530 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:34589
I20260430 20:02:30.831519 17658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:34589 every 8 connection(s)
I20260430 20:02:30.832497 17530 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:30.832952 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17530
I20260430 20:02:30.833089 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:30.833380 10424 external_mini_cluster.cc:1468] Setting key 1fb2a586ebced6d1ab309c43f3f0323e
Apr 30 20:02:30 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:30.845811 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.834619 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:43363 (local address 127.10.46.62:37669)
0430 20:02:30.834817 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:30.834821 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:30.835670 (+ 849us) server_negotiation.cc:408] Connection header received
0430 20:02:30.836673 (+ 1003us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:30.836676 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:30.836725 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:30.836815 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:30.837996 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.838547 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.839202 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.839368 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.842202 (+ 2834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:30.842223 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:30.842226 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:30.842254 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:30.843907 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.844433 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.844436 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.844438 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.844482 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:30.844823 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:30.844826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:30.844828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:30.844982 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:30.845070 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:30.845565 (+ 495us) server_negotiation.cc:300] Negotiation successful
0430 20:02:30.845680 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260430 20:02:30.846493 17661 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:30.835021 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:43363)
0430 20:02:30.835513 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:30.835557 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:30.836468 (+ 911us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:30.836973 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:30.836982 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:30.837370 (+ 388us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:30.837849 (+ 479us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.837861 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.838687 (+ 826us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:30.838691 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:30.839083 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:30.839090 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:30.839246 (+ 156us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:30.840430 (+ 1184us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:30.840450 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:30.842009 (+ 1559us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:30.844028 (+ 2019us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.844034 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.844044 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.844301 (+ 257us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.844593 (+ 292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:30.844596 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:30.844598 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:30.844696 (+ 98us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:30.845081 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:30.845085 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:30.845297 (+ 212us) client_negotiation.cc:770] Sending connection context
0430 20:02:30.845504 (+ 207us) client_negotiation.cc:241] Negotiation successful
0430 20:02:30.845696 (+ 192us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":111,"threads_started":1}
I20260430 20:02:30.847594 17659 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:30.847908 17659 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:30.848450 17659 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:30.849515 17190 ts_manager.cc:194] Registered new tserver with Master: 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:30.850016 17190 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:43363
I20260430 20:02:30.860373 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.46.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:30.916905 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.61:33315
--webserver_interface=127.10.46.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37669,127.10.46.61:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:31.027331 17666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:31.027585 17666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:31.027639 17666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:31.031121 17666 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:31.031189 17666 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:31.031212 17666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:31.031230 17666 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:31.031286 17666 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:31.036092 17666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37669,127.10.46.61:33315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.61:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.46.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
W20260430 20:02:31.036337 17666 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.46.62:37669,127.10.46.61:33315'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260430 20:02:31.037292 17666 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:31.038134 17666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:31.043603 17674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.043615 17671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.043603 17672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:31.044068 17666 server_base.cc:1061] running on GCE node
I20260430 20:02:31.044555 17666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:31.045441 17666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:31.046614 17666 hybrid_clock.cc:648] HybridClock initialized: now 1777579351046600 us; error 26 us; skew 500 ppm
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:31.049679 17666 init.cc:377] Logged in from keytab as oryx/127.10.46.61@KRBTEST.COM (short username oryx)
I20260430 20:02:31.050765 17666 webserver.cc:492] Webserver started at http://127.10.46.61:35791/ using document root <none> and password file <none>
I20260430 20:02:31.051275 17666 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.051321 17666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.051484 17666 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:31.053145 17666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/instance:
uuid: "7ec8f0e25ec749fca187e2322ac4004f"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "51c78109eb04e4e4eb0f5bb04ea73dc2"
server_key_iv: "92441598c526af92790e20b759773245"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.053730 17666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal/instance:
uuid: "7ec8f0e25ec749fca187e2322ac4004f"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "51c78109eb04e4e4eb0f5bb04ea73dc2"
server_key_iv: "92441598c526af92790e20b759773245"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.057193 17666 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:31.059490 17681 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.060532 17666 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:02:31.060686 17666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
uuid: "7ec8f0e25ec749fca187e2322ac4004f"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "51c78109eb04e4e4eb0f5bb04ea73dc2"
server_key_iv: "92441598c526af92790e20b759773245"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.060801 17666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.087793 17666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:31.090773 17666 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:31.090998 17666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:31.098363 17733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.61:33315 every 8 connection(s)
I20260430 20:02:31.098366 17666 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.61:33315
I20260430 20:02:31.099623 17666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/info.pb
I20260430 20:02:31.102602 17734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:31.103633 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17666
I20260430 20:02:31.106856 17734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.61@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:31.117216 17737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.105112 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.10.46.61:33315)
0430 20:02:31.105631 (+ 519us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.105640 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.105684 (+ 44us) server_negotiation.cc:408] Connection header received
0430 20:02:31.106414 (+ 730us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.106431 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.106705 (+ 274us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.107016 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.107948 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.109068 (+ 1120us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.109745 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.110012 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.112743 (+ 2731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.112768 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.112783 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.112812 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.114577 (+ 1765us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.114998 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.115003 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.115007 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.115063 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.115300 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.115303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.115305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.115627 (+ 322us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.115848 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.116179 (+ 331us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.116410 (+ 231us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":182,"threads_started":1}
I20260430 20:02:31.117537 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.108440 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.10.46.62:37669)
0430 20:02:31.108578 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.108602 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.108967 (+ 365us) server_negotiation.cc:408] Connection header received
0430 20:02:31.109060 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.109063 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.109117 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.109204 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.110104 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.110677 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.111639 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.111902 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.113732 (+ 1830us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.113754 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.113759 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.113791 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.115195 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.116311 (+ 1116us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.116314 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.116315 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.116363 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.116621 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.116624 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.116626 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.116820 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.116945 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.117258 (+ 313us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.117394 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 20:02:31.117807 17738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.108354 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51766)
0430 20:02:31.108717 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.108746 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.108892 (+ 146us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.109253 (+ 361us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.109258 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.109666 (+ 408us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.109967 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.109973 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.110811 (+ 838us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.110817 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.111491 (+ 674us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.111504 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.111661 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.112177 (+ 516us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.112195 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.113593 (+ 1398us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.115905 (+ 2312us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.115912 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.115915 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.116141 (+ 226us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.116464 (+ 323us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.116468 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.116470 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.116523 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.116949 (+ 426us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.116957 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.117058 (+ 101us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.117192 (+ 134us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.117622 (+ 430us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20260430 20:02:31.119266 17734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.61@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
I20260430 20:02:31.127051 17738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.120295 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60304)
0430 20:02:31.120412 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.120426 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.120538 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.120764 (+ 226us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.120767 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.120947 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.121109 (+ 162us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.121114 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.121813 (+ 699us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.121821 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.122151 (+ 330us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.122156 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.122277 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.122790 (+ 513us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.122802 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.124062 (+ 1260us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.125774 (+ 1712us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.125777 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.125779 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.125937 (+ 158us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.126258 (+ 321us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.126261 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.126262 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.126308 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.126674 (+ 366us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.126677 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.126745 (+ 68us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.126834 (+ 89us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.126926 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260430 20:02:31.127156 17737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.120041 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.10.46.61:33315)
0430 20:02:31.120181 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.120187 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.120490 (+ 303us) server_negotiation.cc:408] Connection header received
0430 20:02:31.120624 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.120627 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.120673 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.120755 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.121227 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.121701 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.122255 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.122470 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.124179 (+ 1709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.124195 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.124197 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.124218 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.125651 (+ 1433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.126051 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.126054 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.126056 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.126099 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.126429 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.126432 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.126433 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.126578 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.126676 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.126961 (+ 285us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.127055 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 20:02:31.130192 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Bootstrap starting.
I20260430 20:02:31.132834 17734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:31.133562 17734 log.cc:826] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:31.135394 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: No bootstrap required, opened a new log
I20260430 20:02:31.137955 17734 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } }
I20260430 20:02:31.138165 17734 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:31.138260 17734 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec8f0e25ec749fca187e2322ac4004f, State: Initialized, Role: FOLLOWER
I20260430 20:02:31.138814 17734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } }
I20260430 20:02:31.139333 17740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } } }
I20260430 20:02:31.139439 17740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:31.140023 17734 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: configured and running, proceeding with master startup.
W20260430 20:02:31.145047 17751 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:31.145143 17751 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:31.147341 17666 master_runner.cc:428] Detected that this master 7ec8f0e25ec749fca187e2322ac4004f is joining an existing cluster
I20260430 20:02:31.147423 17666 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.46.61:33315
I20260430 20:02:31.155979 17740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:31.156162 17740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } }
I20260430 20:02:31.157171 17740 leader_election.cc:290] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669)
I20260430 20:02:31.157709 17204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ec8f0e25ec749fca187e2322ac4004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd426cc2dce4b249bbd545a803b624b" is_pre_election: true
I20260430 20:02:31.158317 17683 leader_election.cc:304] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7ec8f0e25ec749fca187e2322ac4004f; no voters: 0fd426cc2dce4b249bbd545a803b624b
I20260430 20:02:31.158705 17740 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 20:02:31.158955 17737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.150074 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.10.46.61:33315)
0430 20:02:31.150209 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.150213 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.150579 (+ 366us) server_negotiation.cc:408] Connection header received
0430 20:02:31.150682 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.150686 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.150745 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.150833 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.151656 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.152399 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.153207 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.153756 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.155172 (+ 1416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.155191 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.155193 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.155216 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.157311 (+ 2095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.157799 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.157803 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.157805 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.157845 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.158070 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.158074 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.158075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.158275 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.158346 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.158671 (+ 325us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.158791 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":1280}
I20260430 20:02:31.159360 17758 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.149974 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60318)
0430 20:02:31.150480 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.150494 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.150597 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.150941 (+ 344us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.150945 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.151204 (+ 259us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.151518 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.151525 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.152523 (+ 998us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.152527 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.153091 (+ 564us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.153099 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.153869 (+ 770us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.154478 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.154495 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.154938 (+ 443us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.157453 (+ 2515us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.157458 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.157462 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.157685 (+ 223us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.157937 (+ 252us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.157940 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.157942 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.157988 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.158422 (+ 434us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.158425 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.158495 (+ 70us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.159085 (+ 590us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.159206 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":210,"threads_started":1}
I20260430 20:02:31.172004 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.150066 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.10.46.62:37669)
0430 20:02:31.150319 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.150324 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.150821 (+ 497us) server_negotiation.cc:408] Connection header received
0430 20:02:31.150951 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.150960 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.151005 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.151093 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.152032 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.152529 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.153174 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.153389 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.168428 (+ 15039us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.168454 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.168457 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.168492 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.170230 (+ 1738us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.170840 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.170845 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.170847 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.170902 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.171165 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.171168 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.171170 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.171350 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.171460 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.171710 (+ 250us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.171839 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260430 20:02:31.172021 17757 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.149974 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51776)
0430 20:02:31.150734 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.150743 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.150838 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.151107 (+ 269us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.151110 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.151664 (+ 554us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.151916 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.151922 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.152657 (+ 735us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.152661 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.153060 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.153065 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.153192 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.153789 (+ 597us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.153812 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.168181 (+ 14369us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.170401 (+ 2220us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.170405 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.170415 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.170673 (+ 258us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.171003 (+ 330us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.171006 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.171008 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.171065 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.171448 (+ 383us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.171451 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.171539 (+ 88us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.171657 (+ 118us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.171839 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":143,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.10.46.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.60@KRBTEST.COM" created.
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.62@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
I20260430 20:02:31.187943 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.177427 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60324)
0430 20:02:31.177832 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.177872 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.178027 (+ 155us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.178494 (+ 467us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.178500 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.179051 (+ 551us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.179378 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.179386 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.180382 (+ 996us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.180386 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.180801 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.180808 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.180910 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.181453 (+ 543us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.181466 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.183471 (+ 2005us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.185956 (+ 2485us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.185969 (+ 13us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.185977 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.186316 (+ 339us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.186626 (+ 310us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.186632 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.186637 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.186701 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.187198 (+ 497us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.187208 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.187361 (+ 153us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.187512 (+ 151us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.187692 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260430 20:02:31.188512 17737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.178049 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.10.46.61:33315)
0430 20:02:31.178177 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.178181 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.178195 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:31.178263 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.178266 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.178321 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.178423 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.179514 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.180249 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.180914 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.181116 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.183615 (+ 2499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.183645 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.183648 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.183674 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.185820 (+ 2146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.186459 (+ 639us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.186466 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.186471 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.186527 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.186848 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.186854 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.186858 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.187062 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.187892 (+ 830us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.187971 (+ 79us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.188347 (+ 376us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 20:02:31.190055 17190 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.10.46.61:33315: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f"
member_type: NON_VOTER
last_known_addr {
host: "127.10.46.61"
port: 33315
}
attrs {
promote: true
}
}
dest_uuid: "0fd426cc2dce4b249bbd545a803b624b"
cas_config_opid_index: -1
I20260430 20:02:31.190810 17190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } }
I20260430 20:02:31.193120 17764 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7ec8f0e25ec749fca187e2322ac4004f (127.10.46.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } }
I20260430 20:02:31.194170 17708 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:31.194824 17764 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.10.46.61:33315
I20260430 20:02:31.195060 17764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } }
I20260430 20:02:31.195159 17764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:31.195334 17666 master.cc:562] Master@127.10.46.61:33315 shutting down...
I20260430 20:02:31.195597 17708 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260430 20:02:31.196231 17763 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 20:02:31.198223 17180 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260430 20:02:31.199124 17180 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b -> Peer 7ec8f0e25ec749fca187e2322ac4004f (127.10.46.61:33315): Couldn't send request to peer 7ec8f0e25ec749fca187e2322ac4004f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260430 20:02:31.200317 17666 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:31.200596 17666 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:31.200699 17666 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:31.206071 17666 master.cc:584] Master@127.10.46.61:33315 shutdown complete.
I20260430 20:02:31.206197 17666 master_runner.cc:305] Clearing existing system tablet
I20260430 20:02:31.206570 17666 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.206636 17666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.207897 17666 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:31.208904 17769 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.209146 17666 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.209263 17666 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:31.209300 17666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
uuid: "7ec8f0e25ec749fca187e2322ac4004f"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "51c78109eb04e4e4eb0f5bb04ea73dc2"
server_key_iv: "92441598c526af92790e20b759773245"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.211397 17666 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Deleting tablet data with delete state TABLET_DATA_DELETED
WARNING: no policy specified for HTTP/127.10.46.60@KRBTEST.COM; defaulting to no policy
I20260430 20:02:31.213752 17666 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260430 20:02:31.213845 17666 log.cc:1199] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260430 20:02:31.214700 17666 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Deleting consensus metadata
I20260430 20:02:31.214890 17666 master_runner.cc:315] Copying system tablet from 127.10.46.62:37669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.60@KRBTEST.COM" created.
I20260430 20:02:31.216163 17666 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: tablet copy: Beginning tablet copy session from remote peer at address 127.10.46.62:37669
I20260430 20:02:31.224671 17775 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.216777 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51788)
0430 20:02:31.217060 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.217076 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.217191 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.217858 (+ 667us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.217862 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.218080 (+ 218us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.218353 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.218361 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.219149 (+ 788us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.219153 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.219756 (+ 603us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.219764 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.219910 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.220472 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.220489 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.220885 (+ 396us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.223057 (+ 2172us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.223062 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.223064 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.223289 (+ 225us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.223577 (+ 288us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.223580 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.223583 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.223639 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.224104 (+ 465us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.224108 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.224201 (+ 93us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.224312 (+ 111us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.224461 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260430 20:02:31.224722 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.217418 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.10.46.62:37669)
0430 20:02:31.217574 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.217580 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.217597 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:31.217650 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.217655 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.217724 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.217824 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.218517 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.219034 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.219984 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.220560 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.221024 (+ 464us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.221048 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.221054 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.221088 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.222931 (+ 1843us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.223402 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.223408 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.223412 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.223472 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.223742 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.223746 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.223748 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.224004 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.224113 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.224387 (+ 274us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.224511 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 20:02:31.225276 17214 tablet_copy_service.cc:140] P 0fd426cc2dce4b249bbd545a803b624b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7ec8f0e25ec749fca187e2322ac4004f ({username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:51788)
I20260430 20:02:31.225442 17214 tablet_copy_service.cc:161] P 0fd426cc2dce4b249bbd545a803b624b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7ec8f0e25ec749fca187e2322ac4004f at {username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:51788: session id = 7ec8f0e25ec749fca187e2322ac4004f-00000000000000000000000000000000
I20260430 20:02:31.227046 17214 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 20:02:31.228782 17666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260430 20:02:31.231715 17666 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: tablet copy: Starting download of 0 data blocks...
I20260430 20:02:31.231966 17666 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: tablet copy: Starting download of 1 WAL segments...
I20260430 20:02:31.234261 17666 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:31.236155 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.60:38489
--webserver_interface=127.10.46.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 20:02:31.237300 17214 tablet_copy_service.cc:342] P 0fd426cc2dce4b249bbd545a803b624b: Request end of tablet copy session 7ec8f0e25ec749fca187e2322ac4004f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:51788
I20260430 20:02:31.237469 17214 tablet_copy_service.cc:434] P 0fd426cc2dce4b249bbd545a803b624b: ending tablet copy session 7ec8f0e25ec749fca187e2322ac4004f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7ec8f0e25ec749fca187e2322ac4004f
W20260430 20:02:31.243350 17666 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260430 20:02:31.245942 17666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:31.247853 17778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.253015 17779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.253074 17781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:31.253482 17666 server_base.cc:1061] running on GCE node
I20260430 20:02:31.253677 17666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:31.254030 17666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:31.255187 17666 hybrid_clock.cc:648] HybridClock initialized: now 1777579351255162 us; error 42 us; skew 500 ppm
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:31.257026 17666 init.cc:377] Logged in from keytab as oryx/127.10.46.61@KRBTEST.COM (short username oryx)
I20260430 20:02:31.257575 17666 webserver.cc:492] Webserver started at http://127.10.46.61:36429/ using document root <none> and password file <none>
I20260430 20:02:31.257790 17666 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.257836 17666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.259234 17666 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:31.260330 17788 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.260682 17666 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260430 20:02:31.260759 17666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
uuid: "7ec8f0e25ec749fca187e2322ac4004f"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "51c78109eb04e4e4eb0f5bb04ea73dc2"
server_key_iv: "92441598c526af92790e20b759773245"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.260843 17666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.268077 17666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:31.270128 17666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:31.274874 17666 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.61:33315
I20260430 20:02:31.275485 17666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-1/data/info.pb
I20260430 20:02:31.276001 17840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.61:33315 every 8 connection(s)
I20260430 20:02:31.277295 17841 sys_catalog.cc:263] Verifying existing consensus state
I20260430 20:02:31.278244 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Bootstrap starting.
I20260430 20:02:31.288573 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 20:02:31.288970 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Bootstrap complete.
I20260430 20:02:31.289673 17841 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } }
I20260430 20:02:31.289808 17841 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ec8f0e25ec749fca187e2322ac4004f, State: Initialized, Role: LEARNER
I20260430 20:02:31.289911 17841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } }
I20260430 20:02:31.290263 17844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } }
I20260430 20:02:31.290383 17841 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:31.290429 17844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: LEARNER
I20260430 20:02:31.293865 17855 catalog_manager.cc:1269] Loaded cluster ID: 378f4f0d68054f2184a1aeb75cd4f491
I20260430 20:02:31.293937 17855 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: loading cluster ID for follower catalog manager: success
I20260430 20:02:31.296095 17855 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: acquiring CA information for follower catalog manager: success
I20260430 20:02:31.297300 17855 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260430 20:02:31.357002 17777 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:31.357383 17777 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260430 20:02:31.357210 17387 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
W20260430 20:02:31.357522 17777 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:31.361653 17777 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:31.361773 17777 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:31.362007 17777 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:31.362098 17777 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:31.362136 17777 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:31.367406 17777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.60:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.46.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:31.368614 17777 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:31.369478 17777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:31.375649 17861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.375680 17860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.375680 17863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:31.375875 17777 server_base.cc:1061] running on GCE node
I20260430 20:02:31.376482 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:31.377365 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:31.378600 17777 hybrid_clock.cc:648] HybridClock initialized: now 1777579351378576 us; error 33 us; skew 500 ppm
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:31.381723 17777 init.cc:377] Logged in from keytab as oryx/127.10.46.60@KRBTEST.COM (short username oryx)
I20260430 20:02:31.383116 17777 webserver.cc:492] Webserver started at http://127.10.46.60:43933/ using document root <none> and password file <none>
I20260430 20:02:31.383702 17777 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.383776 17777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.384050 17777 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:31.385762 17777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/instance:
uuid: "77727b4f754f4d9aa38f310a91c2900d"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "7ab2784f8af1afd2a86d85963a2fce18"
server_key_iv: "2e9e4567eb566eead309011bdda172f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.386250 17777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal/instance:
uuid: "77727b4f754f4d9aa38f310a91c2900d"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "7ab2784f8af1afd2a86d85963a2fce18"
server_key_iv: "2e9e4567eb566eead309011bdda172f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.390260 17777 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260430 20:02:31.392755 17870 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.393812 17777 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 20:02:31.393937 17777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
uuid: "77727b4f754f4d9aa38f310a91c2900d"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "7ab2784f8af1afd2a86d85963a2fce18"
server_key_iv: "2e9e4567eb566eead309011bdda172f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.394047 17777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.403877 17777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:31.406909 17777 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:31.407115 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:31.414499 17777 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.60:38489
I20260430 20:02:31.414516 17922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.60:38489 every 8 connection(s)
I20260430 20:02:31.415586 17777 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/info.pb
I20260430 20:02:31.418341 17923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:31.422111 17923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } has no permanent_uuid. Determining permanent_uuid...
I20260430 20:02:31.424046 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 17777
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.60@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
I20260430 20:02:31.434736 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.423892 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.10.46.62:37669)
0430 20:02:31.424037 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.424040 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.424890 (+ 850us) server_negotiation.cc:408] Connection header received
0430 20:02:31.425884 (+ 994us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.425887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.425935 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.426002 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.427234 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.427735 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.428386 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.429089 (+ 703us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.430870 (+ 1781us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.430892 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.430896 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.430928 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.432547 (+ 1619us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.433146 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.433149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.433151 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.433203 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.433530 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.433533 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.433535 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.433736 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.433844 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.434430 (+ 586us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.434530 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 20:02:31.435398 17925 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.424122 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51792)
0430 20:02:31.424751 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.424797 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.425677 (+ 880us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.426075 (+ 398us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.426081 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.426677 (+ 596us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.427085 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.427095 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.427879 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.427883 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.428258 (+ 375us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.428264 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.428467 (+ 203us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.429144 (+ 677us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.429161 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.430671 (+ 1510us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.432678 (+ 2007us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.432687 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.432697 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.433032 (+ 335us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.433303 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.433311 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.433313 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.433399 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.433861 (+ 462us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.433866 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.434132 (+ 266us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.434405 (+ 273us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.434645 (+ 240us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":167,"threads_started":1}
I20260430 20:02:31.435546 17926 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.424925 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.10.46.60:38489)
0430 20:02:31.425228 (+ 303us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.425242 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.425280 (+ 38us) server_negotiation.cc:408] Connection header received
0430 20:02:31.425677 (+ 397us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.425699 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.425916 (+ 217us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.426098 (+ 182us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.426933 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.427725 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.428368 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.428560 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.431328 (+ 2768us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.431375 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.431387 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.431415 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.433574 (+ 2159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.434034 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.434040 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.434042 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.434096 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.434330 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.434336 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.434339 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.434528 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.434722 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.435239 (+ 517us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.435353 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
I20260430 20:02:31.436923 17923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.60@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
I20260430 20:02:31.444939 17925 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.437471 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60330)
0430 20:02:31.437584 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.437598 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.437677 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.438060 (+ 383us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.438062 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.438197 (+ 135us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.438404 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.438409 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.439304 (+ 895us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.439307 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.439733 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.439739 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.439907 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.440388 (+ 481us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.440401 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.441599 (+ 1198us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.443550 (+ 1951us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.443554 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.443556 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.443734 (+ 178us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.444038 (+ 304us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.444045 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.444047 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.444091 (+ 44us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.444429 (+ 338us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.444432 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.444516 (+ 84us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.444619 (+ 103us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.444766 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260430 20:02:31.444941 17927 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.437572 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.10.46.61:33315)
0430 20:02:31.437859 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.437862 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.437875 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:31.437909 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.437912 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.437952 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.438028 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.438509 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.439189 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.439904 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.440039 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.441749 (+ 1710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.441767 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.441770 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.441793 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.443302 (+ 1509us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.443880 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.443884 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.443886 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.443938 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.444178 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.444182 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.444184 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.444333 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.444447 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.444681 (+ 234us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.444796 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260430 20:02:31.445829 17923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.60@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
I20260430 20:02:31.453857 17925 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.446504 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51898)
0430 20:02:31.446624 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.446640 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.446717 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.446939 (+ 222us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.446941 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.447072 (+ 131us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.447225 (+ 153us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.447229 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.447957 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.447960 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.448356 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.448363 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.448459 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.448942 (+ 483us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.448954 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.450434 (+ 1480us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.452195 (+ 1761us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.452198 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.452199 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.452359 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.452643 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.452646 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.452648 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.452695 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.453178 (+ 483us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.453183 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.453315 (+ 132us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.453504 (+ 189us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.453686 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260430 20:02:31.454231 17926 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.446544 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.10.46.60:38489)
0430 20:02:31.446698 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.446701 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.446714 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:31.446793 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.446796 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.446842 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.446915 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.447336 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.447857 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.448475 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.448659 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.450563 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.450594 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.450597 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.450619 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.452093 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.452466 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.452470 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.452472 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.452520 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.452789 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.452792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.452794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.452964 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.453052 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.453580 (+ 528us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.453789 (+ 209us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":80}
I20260430 20:02:31.457086 17923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Bootstrap starting.
I20260430 20:02:31.459422 17923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:31.460170 17923 log.cc:826] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:31.461850 17923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: No bootstrap required, opened a new log
I20260430 20:02:31.464406 17923 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } }
I20260430 20:02:31.464568 17923 raft_consensus.cc:385] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:31.464617 17923 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77727b4f754f4d9aa38f310a91c2900d, State: Initialized, Role: FOLLOWER
I20260430 20:02:31.465024 17923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } }
I20260430 20:02:31.465571 17929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } } }
I20260430 20:02:31.465741 17929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:31.466348 17923 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: configured and running, proceeding with master startup.
W20260430 20:02:31.472245 17940 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:31.472361 17940 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:31.475739 17777 master_runner.cc:428] Detected that this master 77727b4f754f4d9aa38f310a91c2900d is joining an existing cluster
I20260430 20:02:31.475839 17777 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.46.60:38489
I20260430 20:02:31.493026 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.489984 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51810)
0430 20:02:31.490126 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.490144 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.490242 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.490625 (+ 383us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.490629 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.490654 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:31.490826 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.490830 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.491949 (+ 1119us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.491952 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.492662 (+ 710us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.492669 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.492765 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.492809 (+ 44us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.492849 (+ 40us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.492896 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260430 20:02:31.493623 17949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.490059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.10.46.62:37669)
0430 20:02:31.490390 (+ 331us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.490394 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.490409 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:31.490461 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.490464 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.490511 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.490601 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:31.490934 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.491845 (+ 911us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.492781 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.493332 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.493391 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.493439 (+ 48us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.493496 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"spinlock_wait_cycles":3328,"thread_start_us":137,"threads_started":1}
I20260430 20:02:31.497300 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.494815 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60352)
0430 20:02:31.494963 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.494974 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.495031 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.495449 (+ 418us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.495453 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.495475 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:31.495627 (+ 152us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.495633 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.496393 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.496397 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.496956 (+ 559us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.496962 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.497085 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.497098 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.497147 (+ 49us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.497194 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260430 20:02:31.497897 17950 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.494927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.10.46.61:33315)
0430 20:02:31.495213 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.495217 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.495227 (+ 10us) server_negotiation.cc:408] Connection header received
0430 20:02:31.495267 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.495269 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.495317 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.495429 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:31.495740 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.496277 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.497073 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.497552 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.497676 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.497718 (+ 42us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.497758 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260430 20:02:31.507079 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.478039 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.10.46.62:37669)
0430 20:02:31.478211 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.478216 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.479421 (+ 1205us) server_negotiation.cc:408] Connection header received
0430 20:02:31.479547 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.479551 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.479605 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.479694 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.500666 (+ 20972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501410 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.502493 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.502692 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.503527 (+ 835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.503544 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.503548 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.503573 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.505218 (+ 1645us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.505792 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.505796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.505798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.505846 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.506125 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.506129 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.506131 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.506293 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.506380 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.506820 (+ 440us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.506921 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 20:02:31.507599 17948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.478195 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51800)
0430 20:02:31.479317 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.479331 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.479428 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.479716 (+ 288us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.479719 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.500172 (+ 20453us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.500488 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.500498 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501588 (+ 1090us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.501592 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.502181 (+ 589us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.502190 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.502295 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.502904 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.502922 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.503389 (+ 467us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.505361 (+ 1972us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.505364 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.505367 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.505659 (+ 292us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.505939 (+ 280us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.505943 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.505945 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.506015 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.506422 (+ 407us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.506425 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.506505 (+ 80us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.506605 (+ 100us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.507416 (+ 811us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"spinlock_wait_cycles":7296,"thread_start_us":94,"threads_started":1}
I20260430 20:02:31.509634 17947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.478052 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60346)
0430 20:02:31.478483 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.478498 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.478598 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.478927 (+ 329us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.478933 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.500170 (+ 21237us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.500547 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.500559 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501534 (+ 975us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.501538 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.502117 (+ 579us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.502126 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.502233 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.505477 (+ 3244us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.505493 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.505891 (+ 398us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.508229 (+ 2338us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508232 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.508234 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.508427 (+ 193us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.508678 (+ 251us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508682 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.508684 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.508738 (+ 54us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.509107 (+ 369us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.509110 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.509186 (+ 76us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.509293 (+ 107us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.509436 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"spinlock_wait_cycles":13440,"thread_start_us":125,"threads_started":1}
I20260430 20:02:31.509646 17946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.479062 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51914)
0430 20:02:31.479422 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.479435 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.479509 (+ 74us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.479973 (+ 464us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.479975 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.500124 (+ 20149us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.500838 (+ 714us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.500845 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501690 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.501693 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.502199 (+ 506us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.502205 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.502590 (+ 385us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.503074 (+ 484us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.503089 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.503536 (+ 447us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.507745 (+ 4209us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.507748 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.507751 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.508002 (+ 251us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.508233 (+ 231us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508235 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.508237 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.508743 (+ 506us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.509107 (+ 364us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.509110 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.509186 (+ 76us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.509396 (+ 210us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.509478 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":11520,"thread_start_us":119,"threads_started":1}
I20260430 20:02:31.510041 17926 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.478908 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.10.46.60:38489)
0430 20:02:31.479019 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.479023 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.479797 (+ 774us) server_negotiation.cc:408] Connection header received
0430 20:02:31.479833 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.479836 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.479878 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.479953 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.500946 (+ 20993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501567 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.502295 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.502492 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.503665 (+ 1173us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.503680 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.503683 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.503704 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.507618 (+ 3914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508105 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.508108 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.508110 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.508147 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508831 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.508834 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.508836 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.508981 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.509051 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.509347 (+ 296us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.509902 (+ 555us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":438,"server-negotiator.queue_time_us":34}
I20260430 20:02:31.510529 17927 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.478451 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.10.46.61:33315)
0430 20:02:31.478607 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.478613 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.478631 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:02:31.478707 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.478711 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.478763 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.478847 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.500710 (+ 21863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.501417 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.504912 (+ 3495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.505147 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.506051 (+ 904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.506072 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.506075 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.506105 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.508004 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508533 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.508537 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.508539 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.508582 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.508837 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.508841 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.508843 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.508982 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.509068 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.510235 (+ 1167us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.510384 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579349, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.62@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
I20260430 20:02:31.521081 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.512495 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51930)
0430 20:02:31.512632 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.512647 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.512929 (+ 282us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.513304 (+ 375us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.513307 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.513488 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.513678 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.513684 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.514598 (+ 914us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.514602 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.514966 (+ 364us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.514972 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.515078 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.515552 (+ 474us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.515565 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.516856 (+ 1291us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.519490 (+ 2634us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.519493 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.519495 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.519709 (+ 214us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.520009 (+ 300us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.520012 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.520014 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.520080 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.520541 (+ 461us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.520544 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.520631 (+ 87us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.520780 (+ 149us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.520920 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260430 20:02:31.521081 17926 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.512717 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.10.46.60:38489)
0430 20:02:31.512825 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.512830 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.512884 (+ 54us) server_negotiation.cc:408] Connection header received
0430 20:02:31.513051 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.513054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.513103 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.513188 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.513795 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.514473 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.515189 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.515457 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.516984 (+ 1527us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.517003 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.517007 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.517036 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.519265 (+ 2229us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.519853 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.519857 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.519859 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.519910 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.520187 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.520190 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.520191 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.520372 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.520479 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.520833 (+ 354us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.520937 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260430 20:02:31.522329 17194 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.10.46.60:38489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d"
member_type: NON_VOTER
last_known_addr {
host: "127.10.46.60"
port: 38489
}
attrs {
promote: true
}
}
dest_uuid: "0fd426cc2dce4b249bbd545a803b624b"
cas_config_opid_index: 5
I20260430 20:02:31.522866 17194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } }
I20260430 20:02:31.524026 17763 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.524489 17897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:31.525183 17763 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.10.46.60:38489
I20260430 20:02:31.525703 17763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.525786 17763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:31.526261 17777 master.cc:562] Master@127.10.46.60:38489 shutting down...
I20260430 20:02:31.526305 17897 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260430 20:02:31.526808 17815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 LEARNER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260430 20:02:31.527261 17764 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 20:02:31.527699 17764 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:31.529230 17815 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.531080 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd426cc2dce4b249bbd545a803b624b. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.531188 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: LEARNER
I20260430 20:02:31.531286 17844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: NON_VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: true } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.531347 17844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: LEARNER
I20260430 20:02:31.531718 17764 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: attempting to promote NON_VOTER 7ec8f0e25ec749fca187e2322ac4004f to VOTER
I20260430 20:02:31.532225 17764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } }
W20260430 20:02:31.533950 17181 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b -> Peer 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60:38489): Couldn't send request to peer 77727b4f754f4d9aa38f310a91c2900d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260430 20:02:31.534895 17815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 LEARNER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260430 20:02:31.535255 17194 master.cc:680] Remote error: unable to get registration information for peer 77727b4f754f4d9aa38f310a91c2900d: Service unavailable: service kudu.master.MasterService not registered on Master
I20260430 20:02:31.535274 17953 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260430 20:02:31.535851 17777 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:31.536125 17777 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:31.536213 17777 tablet_replica.cc:333] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: stopping tablet replica
I20260430 20:02:31.538683 17763 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ec8f0e25ec749fca187e2322ac4004f (127.10.46.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.539160 17815 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ec8f0e25ec749fca187e2322ac4004f (127.10.46.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.540202 17763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.540334 17763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:31.540488 17763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.540568 17763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:31.540730 17844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.540822 17844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:31.542438 17777 master.cc:584] Master@127.10.46.60:38489 shutdown complete.
I20260430 20:02:31.542481 17777 master_runner.cc:305] Clearing existing system tablet
I20260430 20:02:31.542701 17777 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.542742 17777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.543831 17777 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260430 20:02:31.544638 17955 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.544853 17777 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.544981 17777 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260430 20:02:31.545019 17777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
uuid: "77727b4f754f4d9aa38f310a91c2900d"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "7ab2784f8af1afd2a86d85963a2fce18"
server_key_iv: "2e9e4567eb566eead309011bdda172f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.546584 17777 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:31.548321 17777 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260430 20:02:31.548386 17777 log.cc:1199] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260430 20:02:31.548627 17777 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Deleting consensus metadata
I20260430 20:02:31.548768 17777 master_runner.cc:315] Copying system tablet from 127.10.46.62:37669
I20260430 20:02:31.549948 17777 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: tablet copy: Beginning tablet copy session from remote peer at address 127.10.46.62:37669
I20260430 20:02:31.556553 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.556265 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51944)
0430 20:02:31.556388 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.556457 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.60:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260430 20:02:31.556809 17194 master.cc:680] Network error: unable to get registration information for peer 77727b4f754f4d9aa38f310a91c2900d: Client connection negotiation failed: client connection to 127.10.46.60:38489: connect: Connection refused (error 111)
I20260430 20:02:31.557920 17961 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.550451 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51820)
0430 20:02:31.550784 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.550797 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.550895 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.551150 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.551152 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.551335 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:31.551549 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.551555 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.552653 (+ 1098us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.552656 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.553052 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.553058 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.553152 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.553815 (+ 663us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:31.553828 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:31.554110 (+ 282us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:31.556442 (+ 2332us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.556446 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.556449 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.556665 (+ 216us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.556935 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:31.556937 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:31.556939 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:31.556980 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:31.557382 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:31.557386 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:31.557480 (+ 94us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.557600 (+ 120us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.557742 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":109,"threads_started":1}
I20260430 20:02:31.558008 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.550534 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.10.46.62:37669)
0430 20:02:31.550664 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.550668 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.550882 (+ 214us) server_negotiation.cc:408] Connection header received
0430 20:02:31.550984 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.550988 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.551046 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.551128 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:31.551668 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.552519 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.553160 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.553350 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.554245 (+ 895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:31.554262 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:31.554266 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:31.554291 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:31.556030 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.556786 (+ 756us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.556791 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.556793 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.556847 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:31.557076 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:31.557079 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:31.557081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:31.557278 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:31.557379 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.557771 (+ 392us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.557865 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 20:02:31.558323 17214 tablet_copy_service.cc:140] P 0fd426cc2dce4b249bbd545a803b624b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 77727b4f754f4d9aa38f310a91c2900d ({username='oryx', principal='oryx/127.10.46.60@KRBTEST.COM'} at 127.0.0.1:51820)
I20260430 20:02:31.558414 17214 tablet_copy_service.cc:161] P 0fd426cc2dce4b249bbd545a803b624b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 77727b4f754f4d9aa38f310a91c2900d at {username='oryx', principal='oryx/127.10.46.60@KRBTEST.COM'} at 127.0.0.1:51820: session id = 77727b4f754f4d9aa38f310a91c2900d-00000000000000000000000000000000
I20260430 20:02:31.559401 17214 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 20:02:31.560875 17777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260430 20:02:31.562692 17777 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: tablet copy: Starting download of 0 data blocks...
I20260430 20:02:31.562875 17777 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: tablet copy: Starting download of 1 WAL segments...
I20260430 20:02:31.564746 17777 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 20:02:31.566437 17214 tablet_copy_service.cc:342] P 0fd426cc2dce4b249bbd545a803b624b: Request end of tablet copy session 77727b4f754f4d9aa38f310a91c2900d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.46.60@KRBTEST.COM'} at 127.0.0.1:51820
I20260430 20:02:31.566545 17214 tablet_copy_service.cc:434] P 0fd426cc2dce4b249bbd545a803b624b: ending tablet copy session 77727b4f754f4d9aa38f310a91c2900d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 77727b4f754f4d9aa38f310a91c2900d
W20260430 20:02:31.570824 17777 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260430 20:02:31.573077 17777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:31.575263 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.575459 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:31.575465 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:31.575870 17777 server_base.cc:1061] running on GCE node
I20260430 20:02:31.576081 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:31.576421 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:31.577613 17777 hybrid_clock.cc:648] HybridClock initialized: now 1777579351577600 us; error 33 us; skew 500 ppm
Apr 30 20:02:31 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579351, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:31.579128 17777 init.cc:377] Logged in from keytab as oryx/127.10.46.60@KRBTEST.COM (short username oryx)
I20260430 20:02:31.579648 17777 webserver.cc:492] Webserver started at http://127.10.46.60:41573/ using document root <none> and password file <none>
I20260430 20:02:31.579895 17777 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:31.579947 17777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:31.581030 17777 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260430 20:02:31.582019 17972 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:31.582335 17777 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260430 20:02:31.582464 17777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
uuid: "77727b4f754f4d9aa38f310a91c2900d"
format_stamp: "Formatted at 2026-04-30 20:02:31 on dist-test-slave-0bdw"
server_key: "7ab2784f8af1afd2a86d85963a2fce18"
server_key_iv: "2e9e4567eb566eead309011bdda172f4"
server_key_version: "encryptionkey@0"
I20260430 20:02:31.582562 17777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:31.592944 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.592641 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51954)
0430 20:02:31.592793 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.592851 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.60:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260430 20:02:31.593235 17194 master.cc:680] Network error: unable to get registration information for peer 77727b4f754f4d9aa38f310a91c2900d: Client connection negotiation failed: client connection to 127.10.46.60:38489: connect: Connection refused (error 111)
I20260430 20:02:31.593523 17777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:31.595809 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:31.601889 17777 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.60:38489
I20260430 20:02:31.602073 18024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.60:38489 every 8 connection(s)
I20260430 20:02:31.602517 17777 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/master-2/data/info.pb
I20260430 20:02:31.604233 18025 sys_catalog.cc:263] Verifying existing consensus state
I20260430 20:02:31.605222 18025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Bootstrap starting.
I20260430 20:02:31.608378 17523 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
I20260430 20:02:31.616109 18025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 20:02:31.616428 18025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: Bootstrap complete.
I20260430 20:02:31.617075 18025 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } }
I20260430 20:02:31.617214 18025 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77727b4f754f4d9aa38f310a91c2900d, State: Initialized, Role: LEARNER
I20260430 20:02:31.617321 18025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } }
I20260430 20:02:31.617812 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:31.617928 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: LEARNER
I20260430 20:02:31.617990 18025 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:31.621582 18039 catalog_manager.cc:1269] Loaded cluster ID: 378f4f0d68054f2184a1aeb75cd4f491
I20260430 20:02:31.621642 18039 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: loading cluster ID for follower catalog manager: success
I20260430 20:02:31.623133 18039 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: acquiring CA information for follower catalog manager: success
I20260430 20:02:31.624183 18039 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 20:02:31.664125 17760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.661225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51964)
0430 20:02:31.661382 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:31.661401 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:31.661518 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:31.661913 (+ 395us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:31.661916 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:31.661936 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:31.662133 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.662139 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.663050 (+ 911us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.663055 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:31.663676 (+ 621us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:31.663683 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.663838 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.663857 (+ 19us) client_negotiation.cc:770] Sending connection context
0430 20:02:31.663912 (+ 55us) client_negotiation.cc:241] Negotiation successful
0430 20:02:31.663976 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260430 20:02:31.664624 18040 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:31.661361 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.10.46.60:38489)
0430 20:02:31.661675 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:31.661679 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:31.661691 (+ 12us) server_negotiation.cc:408] Connection header received
0430 20:02:31.661732 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:31.661734 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:31.661781 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:31.661897 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:31.662248 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.662937 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:31.663829 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:31.664305 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:31.664440 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:31.664483 (+ 43us) server_negotiation.cc:300] Negotiation successful
0430 20:02:31.664527 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260430 20:02:31.851799 17659 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
I20260430 20:02:32.063898 17999 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260430 20:02:32.064371 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd426cc2dce4b249bbd545a803b624b. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:32.064512 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: LEARNER
I20260430 20:02:32.530453 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: NON_VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: true } } }
I20260430 20:02:32.530627 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:32.531514 18041 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: attempting to promote NON_VOTER 77727b4f754f4d9aa38f310a91c2900d to VOTER
I20260430 20:02:32.532099 18041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } }
I20260430 20:02:32.533536 17815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260430 20:02:32.533566 17999 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 LEARNER]: Refusing update from remote peer 0fd426cc2dce4b249bbd545a803b624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260430 20:02:32.533986 18042 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260430 20:02:32.534235 18041 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260430 20:02:32.536060 18042 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.536598 17815 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.537395 17999 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.537705 18044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.537797 18044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:32.537744 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.537855 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:32.537920 18044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.537987 18044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:32.538169 18046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.538287 18046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:32.538965 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fd426cc2dce4b249bbd545a803b624b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:32.539052 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:32.586278 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.578302 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.10.46.61:33315)
0430 20:02:32.578549 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.578553 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.578573 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:02:32.578672 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.578675 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.578748 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.578830 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:32.579724 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.580266 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.581002 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.581312 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.582654 (+ 1342us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:32.582673 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:32.582676 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:32.582701 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:32.584598 (+ 1897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.585031 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.585035 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.585037 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.585084 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.585364 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.585367 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.585369 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.585542 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:32.585672 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.585882 (+ 210us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.586015 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260430 20:02:32.590101 18049 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.587285 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51822)
0430 20:02:32.587596 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.587613 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.587716 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.588069 (+ 353us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.588072 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.588092 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.588279 (+ 187us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.588284 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.589190 (+ 906us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.589193 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.589792 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.589799 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.589897 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.589910 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.589948 (+ 38us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.589995 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
I20260430 20:02:32.590711 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.587443 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.10.46.62:37669)
0430 20:02:32.587766 (+ 323us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.587816 (+ 50us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.587832 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:02:32.587884 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.587887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.587942 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.588017 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.588459 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.589075 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.589941 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.590480 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.590513 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.590558 (+ 45us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.590607 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260430 20:02:32.594545 18049 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.592105 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60384)
0430 20:02:32.592227 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.592242 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.592342 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.592614 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.592617 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.592638 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.592819 (+ 181us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.592824 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.593493 (+ 669us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.593498 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.594173 (+ 675us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.594182 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.594293 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.594309 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.594362 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.594418 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"spinlock_wait_cycles":384}
I20260430 20:02:32.594956 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.592238 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.10.46.61:33315)
0430 20:02:32.592342 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.592346 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.592359 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:32.592471 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.592474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.592520 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.592608 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.592915 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.593388 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.594285 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.594734 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.594764 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.594812 (+ 48us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.594879 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":13184}
I20260430 20:02:32.599273 18049 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.596424 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51974)
0430 20:02:32.596819 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.596832 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.596895 (+ 63us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.597216 (+ 321us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.597219 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.597237 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.597396 (+ 159us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.597401 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.598197 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.598200 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.598969 (+ 769us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.598977 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.599085 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.599097 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.599134 (+ 37us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.599179 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260430 20:02:32.599870 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.596619 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.10.46.60:38489)
0430 20:02:32.596891 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.596894 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.596908 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:32.596956 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.596958 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.597000 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.597079 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.597503 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.598070 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.599106 (+ 1036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.599586 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.599617 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.599659 (+ 42us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.599711 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260430 20:02:32.609337 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.601938 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.10.46.60:38489)
0430 20:02:32.602071 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.602077 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.602091 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:32.602252 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.602256 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.602302 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.602387 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:32.603143 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.603614 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.604274 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.604466 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.605747 (+ 1281us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:32.605771 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:32.605774 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:32.605797 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:32.607529 (+ 1732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.608039 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.608043 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.608045 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.608092 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.608513 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.608516 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.608518 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.608690 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:32.608840 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.609053 (+ 213us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.609175 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 20:02:32.613049 18052 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.610200 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:51828)
0430 20:02:32.610580 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.610597 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.610698 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.610939 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.610942 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.610960 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.611191 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.611219 (+ 28us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.612025 (+ 806us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.612029 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.612645 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.612653 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.612787 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.612803 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.612856 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.612910 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260430 20:02:32.613662 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.610380 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.10.46.62:37669)
0430 20:02:32.610519 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.610523 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.610677 (+ 154us) server_negotiation.cc:408] Connection header received
0430 20:02:32.610798 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.610800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.610842 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.610901 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.611329 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.611922 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.612771 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.613276 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.613309 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.613355 (+ 46us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.613571 (+ 216us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 20:02:32.618155 18052 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.615108 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.0.0.1:60398)
0430 20:02:32.615427 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.615441 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.615541 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.615956 (+ 415us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.615959 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.615977 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.616146 (+ 169us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.616151 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.617111 (+ 960us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.617117 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.617742 (+ 625us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.617749 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.617904 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.617920 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.617971 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.618026 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260430 20:02:32.618614 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.615475 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.10.46.61:33315)
0430 20:02:32.615654 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.615657 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.615678 (+ 21us) server_negotiation.cc:408] Connection header received
0430 20:02:32.615739 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.615741 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.615846 (+ 105us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.615931 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.616273 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.616971 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.617890 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.618356 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.618402 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.618477 (+ 75us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.618520 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260430 20:02:32.622227 18052 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.620108 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.0.0.1:51994)
0430 20:02:32.620251 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.620264 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.620330 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.620555 (+ 225us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.620557 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.620570 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:32.620716 (+ 146us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.620720 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.621369 (+ 649us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.621372 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.621947 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.621953 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.622050 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.622061 (+ 11us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.622097 (+ 36us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.622134 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260430 20:02:32.622706 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.620166 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.10.46.60:38489)
0430 20:02:32.620377 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.620388 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.620401 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:32.620433 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.620435 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.620470 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.620536 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:32.620809 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.621268 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.622076 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.622527 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.622554 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.622594 (+ 40us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.622632 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260430 20:02:32.624260 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17258
I20260430 20:02:32.631385 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:45363
--local_ip_for_outbound_sockets=127.10.46.1
--tserver_master_addrs=127.10.46.62:37669
--webserver_port=35689
--webserver_interface=127.10.46.1
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:32.738456 18053 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:32.738698 18053 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:32.738759 18053 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:32.742290 18053 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:32.742409 18053 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:32.742542 18053 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:32.747157 18053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:45363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=35689
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:32.748335 18053 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:32.749210 18053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:32.755357 18059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:32.755623 18058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:32.755638 18061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:32.756202 18053 server_base.cc:1061] running on GCE node
I20260430 20:02:32.756577 18053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:32.757148 18053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:32.758351 18053 hybrid_clock.cc:648] HybridClock initialized: now 1777579352758340 us; error 28 us; skew 500 ppm
Apr 30 20:02:32 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:32.761335 18053 init.cc:377] Logged in from keytab as oryx/127.10.46.1@KRBTEST.COM (short username oryx)
I20260430 20:02:32.762389 18053 webserver.cc:492] Webserver started at http://127.10.46.1:35689/ using document root <none> and password file <none>
I20260430 20:02:32.762917 18053 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:32.762975 18053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:32.766294 18053 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260430 20:02:32.768239 18068 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:32.769255 18053 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:32.769357 18053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "29adfe52affa45fc84188f9ae739e46f"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "8e77950bbb876a578947ea5d56ac645e"
server_key_iv: "68ef09507e36039fed23d89c3a0843cb"
server_key_version: "encryptionkey@0"
I20260430 20:02:32.769750 18053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:32.786972 18053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:32.790130 18053 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:32.790302 18053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:32.790828 18053 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:32.791810 18053 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:32.791923 18053 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:32.791966 18053 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:32.791989 18053 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:32.801664 18053 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:45363
I20260430 20:02:32.801688 18181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:45363 every 8 connection(s)
I20260430 20:02:32.802690 18053 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:32.807791 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18053
I20260430 20:02:32.807984 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17394
Apr 30 20:02:32 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
Apr 30 20:02:32 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
Apr 30 20:02:32 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
I20260430 20:02:32.817409 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:43023
--local_ip_for_outbound_sockets=127.10.46.2
--tserver_master_addrs=127.10.46.62:37669
--webserver_port=32823
--webserver_interface=127.10.46.2
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 20:02:32.819034 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804478 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:42013 (local address 127.10.46.60:38489)
0430 20:02:32.804621 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.804625 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.805423 (+ 798us) server_negotiation.cc:408] Connection header received
0430 20:02:32.806871 (+ 1448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.806875 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.806928 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.807008 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:32.809338 (+ 2330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810070 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.810884 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.811089 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.814401 (+ 3312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:32.814422 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:32.814426 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:32.814457 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:32.816820 (+ 2363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817323 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.817327 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.817329 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.817371 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817728 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.817732 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.817733 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.817918 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:32.817987 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.818740 (+ 753us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.818867 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:02:32.819034 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804848 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:38469 (local address 127.10.46.62:37669)
0430 20:02:32.804998 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.805004 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.806039 (+ 1035us) server_negotiation.cc:408] Connection header received
0430 20:02:32.806592 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.806596 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.806660 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.806716 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:32.808444 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.809188 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.810057 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810260 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.814402 (+ 4142us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:32.814422 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:32.814426 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:32.814457 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:32.816605 (+ 2148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817315 (+ 710us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.817319 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.817320 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.817372 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817873 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.817876 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.817879 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.818056 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:32.818121 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.818740 (+ 619us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.818872 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 20:02:32.820016 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804550 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:45635 (local address 127.10.46.61:33315)
0430 20:02:32.804703 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:32.804707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:32.805440 (+ 733us) server_negotiation.cc:408] Connection header received
0430 20:02:32.806593 (+ 1153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:32.806597 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:32.806665 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:32.806756 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:32.809760 (+ 3004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810569 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.812405 (+ 1836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.812656 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.815586 (+ 2930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:32.815614 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:32.815621 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:32.815662 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:32.817851 (+ 2189us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.818404 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.818411 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.818416 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.818474 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:32.818792 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:32.818799 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:32.818804 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:32.819024 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:32.819111 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:32.819623 (+ 512us) server_negotiation.cc:300] Negotiation successful
0430 20:02:32.819803 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 20:02:32.820488 18188 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804800 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:38469)
0430 20:02:32.805945 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.805960 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.806383 (+ 423us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.806926 (+ 543us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.806934 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.807987 (+ 1053us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:32.808275 (+ 288us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.808287 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.809338 (+ 1051us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.809342 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.809927 (+ 585us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.809934 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810474 (+ 540us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.811428 (+ 954us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:32.811456 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:32.814197 (+ 2741us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:32.816762 (+ 2565us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.816771 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.816788 (+ 17us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.817193 (+ 405us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.817709 (+ 516us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817716 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.817718 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.817765 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.818181 (+ 416us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:32.818184 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:32.818390 (+ 206us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.819567 (+ 1177us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.819669 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":821,"spinlock_wait_cycles":20864,"thread_start_us":102,"threads_started":1}
I20260430 20:02:32.820488 18187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804883 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.10.46.1:42013)
0430 20:02:32.805274 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.805329 (+ 55us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.806383 (+ 1054us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.807032 (+ 649us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.807035 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.807550 (+ 515us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:32.809187 (+ 1637us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.809196 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810192 (+ 996us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.810196 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.810763 (+ 567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.810771 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.810918 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.811449 (+ 531us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:32.811463 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:32.814197 (+ 2734us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:32.816934 (+ 2737us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.816936 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.816938 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.817193 (+ 255us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.817485 (+ 292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.817488 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.817490 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.817628 (+ 138us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.818067 (+ 439us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:32.818072 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:32.818390 (+ 318us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.819278 (+ 888us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.819551 (+ 273us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":77568,"thread_start_us":169,"threads_started":1}
I20260430 20:02:32.820802 18186 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:32.804800 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.10.46.1:45635)
0430 20:02:32.805275 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:32.805330 (+ 55us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:32.806387 (+ 1057us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:32.806938 (+ 551us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:32.806942 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:32.807550 (+ 608us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:32.809601 (+ 2051us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.809613 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.811043 (+ 1430us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:32.811049 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:32.812238 (+ 1189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:32.812250 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:32.812783 (+ 533us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:32.813527 (+ 744us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:32.813552 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:32.815395 (+ 1843us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:32.818012 (+ 2617us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.818018 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.818024 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.818275 (+ 251us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.818603 (+ 328us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:32.818610 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:32.818616 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:32.818682 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:32.819276 (+ 594us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:32.819281 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:32.819389 (+ 108us) client_negotiation.cc:770] Sending connection context
0430 20:02:32.820270 (+ 881us) client_negotiation.cc:241] Negotiation successful
0430 20:02:32.820428 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"spinlock_wait_cycles":16896,"thread_start_us":126,"threads_started":1}
I20260430 20:02:32.822010 18182 heartbeater.cc:344] Connected to a master server at 127.10.46.60:38489
I20260430 20:02:32.822254 18182 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:32.822733 18182 heartbeater.cc:507] Master 127.10.46.60:38489 requested a full tablet report, sending...
I20260430 20:02:32.821986 18184 heartbeater.cc:344] Connected to a master server at 127.10.46.61:33315
I20260430 20:02:32.823263 18184 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:32.821995 18183 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:32.823364 18183 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:32.823467 18184 heartbeater.cc:507] Master 127.10.46.61:33315 requested a full tablet report, sending...
I20260430 20:02:32.823644 18183 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:32.824344 17989 ts_manager.cc:194] Registered new tserver with Master: 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363)
I20260430 20:02:32.824674 17805 ts_manager.cc:194] Registered new tserver with Master: 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363)
I20260430 20:02:32.825978 17194 ts_manager.cc:194] Re-registered known tserver with Master: 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363)
I20260430 20:02:32.826584 17194 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:38469
W20260430 20:02:32.931936 18189 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:32.932216 18189 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:32.932313 18189 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:32.935810 18189 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:32.935913 18189 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:32.936029 18189 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:32.941160 18189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:43023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=32823
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:32.942306 18189 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:32.943147 18189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:32.950387 18194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:32.950405 18195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:32.950443 18197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:32.950786 18189 server_base.cc:1061] running on GCE node
I20260430 20:02:32.951186 18189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:32.951856 18189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:32.953047 18189 hybrid_clock.cc:648] HybridClock initialized: now 1777579352953026 us; error 46 us; skew 500 ppm
Apr 30 20:02:32 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:32.956801 18189 init.cc:377] Logged in from keytab as oryx/127.10.46.2@KRBTEST.COM (short username oryx)
I20260430 20:02:32.958709 18189 webserver.cc:492] Webserver started at http://127.10.46.2:32823/ using document root <none> and password file <none>
I20260430 20:02:32.959407 18189 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:32.959512 18189 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:32.964174 18189 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:32.966609 18204 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:32.968335 18189 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:32.968516 18189 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "a1092265a9534ad5b673c04b278c87c3"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "0527a32cd1e80ec6bc134334ac18e77e"
server_key_iv: "579c42d9b791652cdccb1f4ac9f3ad98"
server_key_version: "encryptionkey@0"
I20260430 20:02:32.968969 18189 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:32.990545 18189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:32.993770 18189 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:32.993984 18189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:32.994550 18189 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:32.995481 18189 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:32.995528 18189 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:32.995579 18189 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:32.995625 18189 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:33.006891 18189 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:43023
I20260430 20:02:33.006948 18317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:43023 every 8 connection(s)
I20260430 20:02:33.007929 18189 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:33.016055 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18189
I20260430 20:02:33.016247 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17530
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579352, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
I20260430 20:02:33.024034 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.010458 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:43409 (local address 127.10.46.61:33315)
0430 20:02:33.010616 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.010621 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.011000 (+ 379us) server_negotiation.cc:408] Connection header received
0430 20:02:33.012614 (+ 1614us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.012618 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.012675 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.012739 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.014378 (+ 1639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.015203 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.016499 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.016757 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.019591 (+ 2834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.019618 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.019622 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.019657 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.021593 (+ 1936us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.022208 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.022215 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.022218 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.022279 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.022695 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.022702 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.022707 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.022920 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.023042 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.023701 (+ 659us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.023848 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 20:02:33.025449 18323 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.010185 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.10.46.2:43409)
0430 20:02:33.010861 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.010901 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.012361 (+ 1460us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.012898 (+ 537us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.012907 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.013387 (+ 480us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.014239 (+ 852us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.014246 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.015749 (+ 1503us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.015797 (+ 48us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.016343 (+ 546us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.016355 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.016887 (+ 532us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.017298 (+ 411us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.017320 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.019388 (+ 2068us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.021710 (+ 2322us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.021718 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.021734 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.022076 (+ 342us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.022399 (+ 323us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.022405 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.022409 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.022541 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.023059 (+ 518us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.023066 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.023383 (+ 317us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.024223 (+ 840us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.024493 (+ 270us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":30,"spinlock_wait_cycles":34432,"thread_start_us":131,"threads_started":1}
I20260430 20:02:33.025558 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.010443 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:50013 (local address 127.10.46.62:37669)
0430 20:02:33.010565 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.010569 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.011517 (+ 948us) server_negotiation.cc:408] Connection header received
0430 20:02:33.012609 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.012614 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.012684 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.012740 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.014234 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.015107 (+ 873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.015944 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.016119 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.021480 (+ 5361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.021508 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.021514 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.021553 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.023571 (+ 2018us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.024288 (+ 717us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.024291 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.024293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.024344 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.024623 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.024626 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.024628 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.024791 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.024867 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.025167 (+ 300us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.025381 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 20:02:33.026105 18324 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.010185 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:50013)
0430 20:02:33.011430 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.011441 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.012363 (+ 922us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.012897 (+ 534us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.012907 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.013387 (+ 480us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.014063 (+ 676us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.014074 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.015222 (+ 1148us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.015225 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.015812 (+ 587us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.015823 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.016249 (+ 426us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.016910 (+ 661us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.016956 (+ 46us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.021294 (+ 4338us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.023955 (+ 2661us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.023959 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.023961 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.024167 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.024452 (+ 285us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.024458 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.024462 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.024511 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.024919 (+ 408us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.024921 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.024995 (+ 74us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.025739 (+ 744us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.025979 (+ 240us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":451,"spinlock_wait_cycles":74368,"thread_start_us":86,"threads_started":1}
I20260430 20:02:33.026695 18320 heartbeater.cc:344] Connected to a master server at 127.10.46.61:33315
I20260430 20:02:33.026799 18319 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:33.026990 18322 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.010185 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.10.46.2:44233)
0430 20:02:33.011061 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.011073 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.012358 (+ 1285us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.012897 (+ 539us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.012905 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.013382 (+ 477us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.014049 (+ 667us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.014065 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.014973 (+ 908us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.014979 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.015534 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.015542 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.016216 (+ 674us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.017008 (+ 792us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.017027 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.020912 (+ 3885us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.024494 (+ 3582us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.024501 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.024515 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.024767 (+ 252us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.025056 (+ 289us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.025062 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.025067 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.025140 (+ 73us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.026409 (+ 1269us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.026412 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.026497 (+ 85us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.026610 (+ 113us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.026745 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"spinlock_wait_cycles":6784,"thread_start_us":139,"threads_started":1}
I20260430 20:02:33.026953 18320 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.026953 18319 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.027243 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.009845 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:44233 (local address 127.10.46.60:38489)
0430 20:02:33.010142 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.010146 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.011157 (+ 1011us) server_negotiation.cc:408] Connection header received
0430 20:02:33.012569 (+ 1412us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.012573 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.012634 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.012734 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.014306 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.014842 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.015680 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.015907 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.021065 (+ 5158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.021091 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.021097 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.021136 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.024355 (+ 3219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.024883 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.024890 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.024894 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.024948 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.025239 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.025245 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.025249 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.025453 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.026321 (+ 868us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.026700 (+ 379us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.026905 (+ 205us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260430 20:02:33.027619 18319 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:33.027621 18320 heartbeater.cc:507] Master 127.10.46.61:33315 requested a full tablet report, sending...
I20260430 20:02:33.028227 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:34589
--local_ip_for_outbound_sockets=127.10.46.3
--tserver_master_addrs=127.10.46.62:37669
--webserver_port=39809
--webserver_interface=127.10.46.3
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489 with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 20:02:33.028599 18318 heartbeater.cc:344] Connected to a master server at 127.10.46.60:38489
I20260430 20:02:33.028725 18318 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.028875 17194 ts_manager.cc:194] Re-registered known tserver with Master: a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
I20260430 20:02:33.029022 18318 heartbeater.cc:507] Master 127.10.46.60:38489 requested a full tablet report, sending...
I20260430 20:02:33.029582 17194 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:50013
I20260430 20:02:33.029898 17989 ts_manager.cc:194] Registered new tserver with Master: a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
I20260430 20:02:33.030210 17805 ts_manager.cc:194] Registered new tserver with Master: a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
W20260430 20:02:33.139528 18325 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:33.139853 18325 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:33.139920 18325 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:33.143684 18325 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:33.143834 18325 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:33.143941 18325 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:33.148887 18325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:34589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=39809
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37669,127.10.46.61:33315,127.10.46.60:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:33.150051 18325 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:33.150913 18325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:33.157696 18333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:33.157696 18331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:33.157774 18330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:33.157958 18325 server_base.cc:1061] running on GCE node
I20260430 20:02:33.158435 18325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:33.159029 18325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:33.160238 18325 hybrid_clock.cc:648] HybridClock initialized: now 1777579353160228 us; error 25 us; skew 500 ppm
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579353, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:33.163215 18325 init.cc:377] Logged in from keytab as oryx/127.10.46.3@KRBTEST.COM (short username oryx)
I20260430 20:02:33.164288 18325 webserver.cc:492] Webserver started at http://127.10.46.3:39809/ using document root <none> and password file <none>
I20260430 20:02:33.164886 18325 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:33.164949 18325 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:33.168468 18325 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:02:33.170275 18340 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:33.171386 18325 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:33.171520 18325 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "63ee5bda423347388be745e1042e726d"
format_stamp: "Formatted at 2026-04-30 20:02:30 on dist-test-slave-0bdw"
server_key: "35988facc1e4fcfb811ab669d9da1814"
server_key_iv: "60cf062a750a2acdb9ccd2d51a5cdb0f"
server_key_version: "encryptionkey@0"
I20260430 20:02:33.171983 18325 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:33.216625 18325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:33.220325 18325 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:33.220825 18325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:33.221596 18325 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:33.222589 18325 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:33.222642 18325 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:33.222682 18325 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:33.222698 18325 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:33.233606 18325 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:34589
I20260430 20:02:33.233628 18453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:34589 every 8 connection(s)
I20260430 20:02:33.234751 18325 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:33.235724 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18325
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579353, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.60@KRBTEST.COM
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579353, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:33.249315 18468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.238903 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.10.46.60:38489)
0430 20:02:33.239522 (+ 619us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.239525 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.239818 (+ 293us) server_negotiation.cc:408] Connection header received
0430 20:02:33.239915 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.239921 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.239979 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.240060 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.241691 (+ 1631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.242506 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.243798 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.244021 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.245305 (+ 1284us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.245329 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.245333 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.245366 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.247416 (+ 2050us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.247893 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.247896 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.247898 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.247945 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.248169 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.248172 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.248174 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.248346 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.248485 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.249004 (+ 519us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.249129 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":62,"threads_started":1}
I20260430 20:02:33.251986 18466 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.238837 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.10.46.62:37669)
0430 20:02:33.239153 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.239157 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.239520 (+ 363us) server_negotiation.cc:408] Connection header received
0430 20:02:33.239574 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.239577 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.239633 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.239740 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.241671 (+ 1931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.242506 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.244025 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.244233 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.247010 (+ 2777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.247100 (+ 90us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.247105 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.247138 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.249339 (+ 2201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250372 (+ 1033us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.250376 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.250378 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.250451 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250745 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.250749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.250751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.250933 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.251057 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.251670 (+ 613us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.251814 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260430 20:02:33.252161 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.237333 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:39473 (local address 127.10.46.60:38489)
0430 20:02:33.237607 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.237619 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.238332 (+ 713us) server_negotiation.cc:408] Connection header received
0430 20:02:33.240188 (+ 1856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.240192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.240239 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.240289 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.242127 (+ 1838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.242898 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.243802 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.243995 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.246917 (+ 2922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.246936 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.246940 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.246967 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.249038 (+ 2071us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250404 (+ 1366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.250409 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.250414 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.250465 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250854 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.250858 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.250860 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.251025 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.251092 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.251843 (+ 751us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.251953 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"spinlock_wait_cycles":5376}
I20260430 20:02:33.253527 18464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.237032 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.60:38489 (local address 127.10.46.3:39473)
0430 20:02:33.238200 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.238244 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.239970 (+ 1726us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.240597 (+ 627us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.240609 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.241285 (+ 676us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.241955 (+ 670us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.241971 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.243041 (+ 1070us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.243044 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.243644 (+ 600us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.243651 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.244235 (+ 584us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.244747 (+ 512us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.244776 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.246726 (+ 1950us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.249870 (+ 3144us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.249880 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.249896 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.250272 (+ 376us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.250595 (+ 323us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250600 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.250603 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.250736 (+ 133us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.251176 (+ 440us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.251182 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.251493 (+ 311us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.252332 (+ 839us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.252600 (+ 268us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"spinlock_wait_cycles":5504,"thread_start_us":117,"threads_started":1}
I20260430 20:02:33.254745 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.238120 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:37201 (local address 127.10.46.62:37669)
0430 20:02:33.238349 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.238356 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.238497 (+ 141us) server_negotiation.cc:408] Connection header received
0430 20:02:33.240284 (+ 1787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.240287 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.240463 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.240543 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.242561 (+ 2018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.243339 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.244210 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.244418 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.247542 (+ 3124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.247564 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.247570 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.247600 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.250874 (+ 3274us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.251546 (+ 672us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.251550 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.251552 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.251598 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.253326 (+ 1728us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.253329 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.253331 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.253526 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.253604 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.254499 (+ 895us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.254609 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"spinlock_wait_cycles":256768}
I20260430 20:02:33.255195 18463 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.237032 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:37201)
0430 20:02:33.238202 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.238244 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.239969 (+ 1725us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.240683 (+ 714us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.240687 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.242121 (+ 1434us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.242424 (+ 303us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.242430 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.243469 (+ 1039us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.243472 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.244078 (+ 606us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.244088 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.244530 (+ 442us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.245701 (+ 1171us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.245718 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.247400 (+ 1682us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.250995 (+ 3595us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.250998 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.251000 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.251206 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.253158 (+ 1952us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.253161 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.253164 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.253218 (+ 54us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.253683 (+ 465us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.253688 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.254283 (+ 595us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.254877 (+ 594us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.255022 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":27,"spinlock_wait_cycles":1155328,"thread_start_us":94,"threads_started":1}
I20260430 20:02:33.255371 18454 heartbeater.cc:344] Connected to a master server at 127.10.46.60:38489
I20260430 20:02:33.255609 18454 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.256139 18465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.238432 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.10.46.61:33315)
0430 20:02:33.241359 (+ 2927us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.241364 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.241379 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:33.241434 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.241437 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.241481 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.241575 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.247439 (+ 5864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.248252 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.249112 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.249293 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.251044 (+ 1751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.251066 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.251072 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.251103 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.253418 (+ 2315us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.254066 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.254070 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.254071 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.254121 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.255053 (+ 932us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.255057 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.255059 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.255225 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.255494 (+ 269us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.255870 (+ 376us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.255988 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":62,"threads_started":1}
I20260430 20:02:33.256239 18454 heartbeater.cc:507] Master 127.10.46.60:38489 requested a full tablet report, sending...
Apr 30 20:02:33 dist-test-slave-0bdw krb5kdc[17146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579353, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.61@KRBTEST.COM
I20260430 20:02:33.257177 18455 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37669
I20260430 20:02:33.257253 18455 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.257478 18455 heartbeater.cc:507] Master 127.10.46.62:37669 requested a full tablet report, sending...
I20260430 20:02:33.255601 17194 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 20:02:33.258169 17190 ts_manager.cc:194] Re-registered known tserver with Master: 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:33.258227 17989 ts_manager.cc:194] Registered new tserver with Master: 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:33.258859 17190 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:37201
W20260430 20:02:33.259075 17194 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:33.260748 18462 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.237032 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.61:33315 (local address 127.10.46.3:54985)
0430 20:02:33.239178 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.239192 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.239971 (+ 779us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.240597 (+ 626us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.240609 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.241289 (+ 680us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:33.241955 (+ 666us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.241971 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.251431 (+ 9460us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.251434 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.252026 (+ 592us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.252413 (+ 387us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.252843 (+ 430us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.254239 (+ 1396us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:33.254259 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:33.256702 (+ 2443us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:33.259037 (+ 2335us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.259041 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.259043 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.259251 (+ 208us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.259544 (+ 293us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:33.259548 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:33.259551 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:33.259617 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:33.260234 (+ 617us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:33.260237 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:33.260319 (+ 82us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.260434 (+ 115us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.260568 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"mutex_wait_us":775,"spinlock_wait_cycles":14848,"thread_start_us":102,"threads_started":1}
I20260430 20:02:33.261035 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.237463 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:54985 (local address 127.10.46.61:33315)
0430 20:02:33.237795 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.237801 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.239273 (+ 1472us) server_negotiation.cc:408] Connection header received
0430 20:02:33.240190 (+ 917us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.240193 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.240239 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.240300 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.244812 (+ 4512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.245544 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.252539 (+ 6995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.252737 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.256843 (+ 4106us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.256861 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.256867 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.256895 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.258909 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.259366 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.259370 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.259374 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.259424 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.259796 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.259800 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.259803 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.259962 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.260284 (+ 322us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.260659 (+ 375us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.260774 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260430 20:02:33.262223 18456 heartbeater.cc:344] Connected to a master server at 127.10.46.61:33315
I20260430 20:02:33.262336 18456 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:33.262508 18456 heartbeater.cc:507] Master 127.10.46.61:33315 requested a full tablet report, sending...
I20260430 20:02:33.263298 17805 ts_manager.cc:194] Registered new tserver with Master: 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:33.278730 18481 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.274203 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:34589 (local address 127.0.0.1:55710)
0430 20:02:33.274511 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.274526 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.274631 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.275521 (+ 890us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.275525 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.275545 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.275822 (+ 277us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.275829 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277341 (+ 1512us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.277345 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.278341 (+ 996us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.278349 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.278446 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.278462 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.278503 (+ 41us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.278556 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":140,"threads_started":1}
I20260430 20:02:33.278743 18478 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.272790 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:45363 (local address 127.0.0.1:60728)
0430 20:02:33.273246 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.273262 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.273387 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.275269 (+ 1882us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.275272 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.275292 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.275588 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.275595 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277421 (+ 1826us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.277425 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.278348 (+ 923us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.278357 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.278445 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.278466 (+ 21us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.278508 (+ 42us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.278576 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":276,"threads_started":1}
I20260430 20:02:33.278743 18480 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.273618 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:43023 (local address 127.0.0.1:57344)
0430 20:02:33.274025 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.274044 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.274283 (+ 239us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.275578 (+ 1295us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.275581 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.275599 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.275887 (+ 288us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.275898 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277617 (+ 1719us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.277621 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.278428 (+ 807us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.278434 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.278502 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.278515 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.278563 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.278603 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":162,"threads_started":1}
I20260430 20:02:33.279897 18479 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.272933 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.10.46.1:45363)
0430 20:02:33.274747 (+ 1814us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.274755 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.274781 (+ 26us) server_negotiation.cc:408] Connection header received
0430 20:02:33.274876 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.274883 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.275094 (+ 211us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.275737 (+ 643us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.275916 (+ 179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277235 (+ 1319us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.278849 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.279443 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.279568 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.279636 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.279724 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1611,"thread_start_us":117,"threads_started":1}
I20260430 20:02:33.280391 18483 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.274800 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.10.46.2:43023)
0430 20:02:33.275200 (+ 400us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.275207 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.275228 (+ 21us) server_negotiation.cc:408] Connection header received
0430 20:02:33.275299 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.275303 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.275431 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.276028 (+ 597us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.276201 (+ 173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277470 (+ 1269us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.279320 (+ 1850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.279843 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.279930 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.280168 (+ 238us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.280247 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
I20260430 20:02:33.281205 18482 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.274713 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.10.46.3:34589)
0430 20:02:33.275020 (+ 307us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.275027 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.275046 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:02:33.275119 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.275124 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.275355 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.275952 (+ 597us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.276081 (+ 129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.277211 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.279671 (+ 2460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.280837 (+ 1166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.280938 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.280995 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.281069 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260430 20:02:33.282931 18252 tablet_service.cc:1511] Processing CreateTablet for tablet a6db5dee18e745058d104efe5642236f (DEFAULT_TABLE table=test-table [id=f18084b74c1c4cf9bbfcfda08502b665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:33.282931 18116 tablet_service.cc:1511] Processing CreateTablet for tablet a6db5dee18e745058d104efe5642236f (DEFAULT_TABLE table=test-table [id=f18084b74c1c4cf9bbfcfda08502b665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:33.283437 18388 tablet_service.cc:1511] Processing CreateTablet for tablet a6db5dee18e745058d104efe5642236f (DEFAULT_TABLE table=test-table [id=f18084b74c1c4cf9bbfcfda08502b665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:33.283948 18116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6db5dee18e745058d104efe5642236f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:33.283948 18252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6db5dee18e745058d104efe5642236f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:33.284257 18388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6db5dee18e745058d104efe5642236f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:33.291523 18485 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Bootstrap starting.
I20260430 20:02:33.292258 18484 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Bootstrap starting.
I20260430 20:02:33.292434 18486 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Bootstrap starting.
I20260430 20:02:33.294358 18484 tablet_bootstrap.cc:654] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:33.294358 18486 tablet_bootstrap.cc:654] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:33.295056 18484 log.cc:826] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:33.295056 18486 log.cc:826] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:33.297060 18484 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: No bootstrap required, opened a new log
I20260430 20:02:33.297158 18486 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: No bootstrap required, opened a new log
I20260430 20:02:33.297236 18485 tablet_bootstrap.cc:654] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:33.297349 18486 ts_tablet_manager.cc:1403] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 20:02:33.297441 18484 ts_tablet_manager.cc:1403] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:02:33.298024 18485 log.cc:826] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:33.299521 18485 tablet_bootstrap.cc:492] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: No bootstrap required, opened a new log
I20260430 20:02:33.299731 18485 ts_tablet_manager.cc:1403] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260430 20:02:33.300572 18486 raft_consensus.cc:359] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.300572 18484 raft_consensus.cc:359] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.300782 18486 raft_consensus.cc:385] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:33.300782 18484 raft_consensus.cc:385] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:33.300841 18484 raft_consensus.cc:740] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1092265a9534ad5b673c04b278c87c3, State: Initialized, Role: FOLLOWER
I20260430 20:02:33.300841 18486 raft_consensus.cc:740] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee5bda423347388be745e1042e726d, State: Initialized, Role: FOLLOWER
I20260430 20:02:33.301268 18486 consensus_queue.cc:260] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.301268 18484 consensus_queue.cc:260] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.302153 18484 ts_tablet_manager.cc:1434] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:02:33.302153 18486 ts_tablet_manager.cc:1434] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:02:33.302368 18319 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
I20260430 20:02:33.303385 18455 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
I20260430 20:02:33.304899 18485 raft_consensus.cc:359] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.305143 18485 raft_consensus.cc:385] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:33.305214 18485 raft_consensus.cc:740] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29adfe52affa45fc84188f9ae739e46f, State: Initialized, Role: FOLLOWER
I20260430 20:02:33.305757 18485 consensus_queue.cc:260] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.306749 18485 ts_tablet_manager.cc:1434] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260430 20:02:33.307106 18183 heartbeater.cc:499] Master 127.10.46.62:37669 was elected leader, sending a full tablet report...
W20260430 20:02:33.493165 18457 tablet.cc:2404] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:02:33.509377 18321 tablet.cc:2404] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:02:33.555591 18185 tablet.cc:2404] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:02:33.610075 18491 raft_consensus.cc:493] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:33.610291 18491 raft_consensus.cc:515] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.612527 18491 leader_election.cc:290] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363), a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
I20260430 20:02:33.614251 18492 raft_consensus.cc:493] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:33.614415 18492 raft_consensus.cc:515] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.616427 18492 leader_election.cc:290] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023), 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:33.618053 18463 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.612210 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:43023 (local address 127.10.46.3:38989)
0430 20:02:33.612350 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.612362 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.612444 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.614952 (+ 2508us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.614958 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.614988 (+ 30us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.615267 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.615277 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.616799 (+ 1522us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.616803 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.617702 (+ 899us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.617710 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.617800 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.617817 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.617865 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.617926 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260430 20:02:33.619594 18493 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.615807 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:43023 (local address 127.10.46.1:55761)
0430 20:02:33.616055 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.616072 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.616174 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.617037 (+ 863us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.617040 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.617084 (+ 44us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.617308 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.617315 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.618270 (+ 955us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.618273 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.619168 (+ 895us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.619177 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.619317 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.619333 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.619381 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.619436 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"spinlock_wait_cycles":10624,"thread_start_us":73,"threads_started":1}
I20260430 20:02:33.619674 18462 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.611850 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:45363 (local address 127.10.46.3:36685)
0430 20:02:33.612134 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.612154 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.612275 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.613251 (+ 976us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.613254 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.613272 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.613545 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.613551 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.615006 (+ 1455us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.615011 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.619309 (+ 4298us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.619317 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.619430 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.619443 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.619491 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.619540 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260430 20:02:33.620631 18479 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.611953 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:36685 (local address 127.10.46.1:45363)
0430 20:02:33.612945 (+ 992us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.612949 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.612966 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:33.613023 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.613026 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.613075 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.613150 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.613673 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.614390 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.619941 (+ 5551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.620414 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.620440 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.620483 (+ 43us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.620531 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895}
I20260430 20:02:33.620678 18494 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.615907 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:55761 (local address 127.10.46.2:43023)
0430 20:02:33.616812 (+ 905us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.616816 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.616827 (+ 11us) server_negotiation.cc:408] Connection header received
0430 20:02:33.616867 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.616871 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.616916 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.617414 (+ 498us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.617496 (+ 82us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.618151 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.619973 (+ 1822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.620408 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.620435 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.620483 (+ 48us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.620535 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":68,"threads_started":1}
I20260430 20:02:33.620802 18482 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.616336 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:49857 (local address 127.10.46.3:34589)
0430 20:02:33.616593 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.616598 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.616723 (+ 125us) server_negotiation.cc:408] Connection header received
0430 20:02:33.616878 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.616882 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.616923 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.616996 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.617546 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.618314 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.619694 (+ 1380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.620233 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.620264 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.620592 (+ 328us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.620658 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260430 20:02:33.621197 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "63ee5bda423347388be745e1042e726d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29adfe52affa45fc84188f9ae739e46f" is_pre_election: true
I20260430 20:02:33.621335 18272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "29adfe52affa45fc84188f9ae739e46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1092265a9534ad5b673c04b278c87c3" is_pre_election: true
I20260430 20:02:33.621441 18136 raft_consensus.cc:2468] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee5bda423347388be745e1042e726d in term 0.
I20260430 20:02:33.621553 18272 raft_consensus.cc:2468] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29adfe52affa45fc84188f9ae739e46f in term 0.
I20260430 20:02:33.621603 18495 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.616270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:34589 (local address 127.10.46.1:49857)
0430 20:02:33.616650 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:33.616667 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:33.616775 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:33.617033 (+ 258us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:33.617036 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:33.617088 (+ 52us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:33.617411 (+ 323us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.617423 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.618460 (+ 1037us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.618465 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:33.619505 (+ 1040us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:33.619515 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.620431 (+ 916us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.620451 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:02:33.621129 (+ 678us) client_negotiation.cc:241] Negotiation successful
0430 20:02:33.621194 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"spinlock_wait_cycles":2944,"thread_start_us":55,"threads_started":1}
I20260430 20:02:33.621891 18344 leader_election.cc:304] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29adfe52affa45fc84188f9ae739e46f, 63ee5bda423347388be745e1042e726d; no voters:
I20260430 20:02:33.621917 18072 leader_election.cc:304] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29adfe52affa45fc84188f9ae739e46f, a1092265a9534ad5b673c04b278c87c3; no voters:
I20260430 20:02:33.622264 18492 raft_consensus.cc:2804] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:33.622339 18492 raft_consensus.cc:493] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:33.622367 18483 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.614578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:38989 (local address 127.10.46.2:43023)
0430 20:02:33.614707 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.614711 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.614725 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:33.614773 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.614776 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.614823 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.614903 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:33.615856 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.616544 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.618146 (+ 1602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.619168 (+ 1022us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.622128 (+ 2960us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.622183 (+ 55us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.622255 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 20:02:33.622381 18492 raft_consensus.cc:3060] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:33.622704 18491 raft_consensus.cc:2804] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:33.622773 18491 raft_consensus.cc:493] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:33.622814 18491 raft_consensus.cc:3060] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:33.622766 18272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "63ee5bda423347388be745e1042e726d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1092265a9534ad5b673c04b278c87c3" is_pre_election: true
I20260430 20:02:33.622902 18272 raft_consensus.cc:2468] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee5bda423347388be745e1042e726d in term 0.
I20260430 20:02:33.623842 18492 raft_consensus.cc:515] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.624243 18492 leader_election.cc:290] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [CANDIDATE]: Term 1 election: Requested vote from peers a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023), 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:33.624562 18491 raft_consensus.cc:515] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.624897 18408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "29adfe52affa45fc84188f9ae739e46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee5bda423347388be745e1042e726d" is_pre_election: true
I20260430 20:02:33.625082 18408 raft_consensus.cc:2393] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29adfe52affa45fc84188f9ae739e46f in current term 1: Already voted for candidate 63ee5bda423347388be745e1042e726d in this term.
I20260430 20:02:33.625545 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "63ee5bda423347388be745e1042e726d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29adfe52affa45fc84188f9ae739e46f"
I20260430 20:02:33.625718 18136 raft_consensus.cc:2393] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 63ee5bda423347388be745e1042e726d in current term 1: Already voted for candidate 29adfe52affa45fc84188f9ae739e46f in this term.
I20260430 20:02:33.626156 18408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "29adfe52affa45fc84188f9ae739e46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee5bda423347388be745e1042e726d"
I20260430 20:02:33.626228 18272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "63ee5bda423347388be745e1042e726d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1092265a9534ad5b673c04b278c87c3"
I20260430 20:02:33.626286 18408 raft_consensus.cc:2393] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 29adfe52affa45fc84188f9ae739e46f in current term 1: Already voted for candidate 63ee5bda423347388be745e1042e726d in this term.
I20260430 20:02:33.626304 18271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6db5dee18e745058d104efe5642236f" candidate_uuid: "29adfe52affa45fc84188f9ae739e46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1092265a9534ad5b673c04b278c87c3"
I20260430 20:02:33.626346 18272 raft_consensus.cc:3060] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:33.626756 18072 leader_election.cc:304] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29adfe52affa45fc84188f9ae739e46f; no voters: 63ee5bda423347388be745e1042e726d, a1092265a9534ad5b673c04b278c87c3
I20260430 20:02:33.626962 18492 raft_consensus.cc:2749] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 20:02:33.627497 18272 raft_consensus.cc:2468] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee5bda423347388be745e1042e726d in term 1.
I20260430 20:02:33.627668 18491 leader_election.cc:290] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [CANDIDATE]: Term 1 election: Requested vote from peers 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363), a1092265a9534ad5b673c04b278c87c3 (127.10.46.2:43023)
I20260430 20:02:33.627828 18344 leader_election.cc:304] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63ee5bda423347388be745e1042e726d, a1092265a9534ad5b673c04b278c87c3; no voters: 29adfe52affa45fc84188f9ae739e46f
I20260430 20:02:33.628043 18491 raft_consensus.cc:2804] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:33.628268 18491 raft_consensus.cc:697] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 LEADER]: Becoming Leader. State: Replica: 63ee5bda423347388be745e1042e726d, State: Running, Role: LEADER
I20260430 20:02:33.628626 18491 consensus_queue.cc:237] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:33.632030 17194 catalog_manager.cc:5671] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee5bda423347388be745e1042e726d (127.10.46.3). New cstate: current_term: 1 leader_uuid: "63ee5bda423347388be745e1042e726d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } health_report { overall_health: HEALTHY } } }
I20260430 20:02:33.651137 18482 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.648112 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.10.46.3:34589)
0430 20:02:33.648248 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.648252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.648265 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:33.648340 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.648342 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.648412 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.648496 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:33.648873 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.649364 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.650123 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.650293 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.650376 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:33.650754 (+ 378us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:33.650852 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.650977 (+ 125us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.651024 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 20:02:33.656646 18136 raft_consensus.cc:1275] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Refusing update from remote peer 63ee5bda423347388be745e1042e726d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:33.656692 18272 raft_consensus.cc:1275] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Refusing update from remote peer 63ee5bda423347388be745e1042e726d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:33.657277 18491 consensus_queue.cc:1048] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:33.657491 18496 consensus_queue.cc:1048] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:33.668892 18500 mvcc.cc:204] Tried to move back new op lower bound from 7280965032570621952 to 7280965032465555456. Current Snapshot: MvccSnapshot[applied={T|T < 7280965032570621952}]
I20260430 20:02:33.668982 18501 mvcc.cc:204] Tried to move back new op lower bound from 7280965032570621952 to 7280965032465555456. Current Snapshot: MvccSnapshot[applied={T|T < 7280965032570621952}]
I20260430 20:02:33.675979 17190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:33.676177 17190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:33.681491 17190 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 0fd426cc2dce4b249bbd545a803b624b: Sending DeleteTablet for 3 replicas of tablet a6db5dee18e745058d104efe5642236f
I20260430 20:02:33.682283 18388 tablet_service.cc:1558] Processing DeleteTablet for tablet a6db5dee18e745058d104efe5642236f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:33 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:55710
I20260430 20:02:33.682353 18116 tablet_service.cc:1558] Processing DeleteTablet for tablet a6db5dee18e745058d104efe5642236f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:33 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:60728
I20260430 20:02:33.682546 18252 tablet_service.cc:1558] Processing DeleteTablet for tablet a6db5dee18e745058d104efe5642236f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:33 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:57344
I20260430 20:02:33.684111 18512 tablet_replica.cc:333] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: stopping tablet replica
I20260430 20:02:33.684234 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17162
I20260430 20:02:33.684957 18512 raft_consensus.cc:2243] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:33.685322 18512 raft_consensus.cc:2272] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:33.686787 18512 ts_tablet_manager.cc:1916] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:33.688376 18511 tablet_replica.cc:333] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: stopping tablet replica
I20260430 20:02:33.688769 18511 raft_consensus.cc:2243] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:33.689195 18511 raft_consensus.cc:2272] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:33.689568 18512 ts_tablet_manager.cc:1929] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:33.689674 18512 log.cc:1199] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/a6db5dee18e745058d104efe5642236f
I20260430 20:02:33.689929 18510 tablet_replica.cc:333] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: stopping tablet replica
I20260430 20:02:33.690003 18512 ts_tablet_manager.cc:1950] T a6db5dee18e745058d104efe5642236f P a1092265a9534ad5b673c04b278c87c3: Deleting consensus metadata
I20260430 20:02:33.690292 18510 raft_consensus.cc:2243] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:33.690752 18510 raft_consensus.cc:2272] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:33.692147 18510 ts_tablet_manager.cc:1916] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:33.692305 18511 ts_tablet_manager.cc:1916] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260430 20:02:33.693830 18208 connection.cc:570] server connection from 127.0.0.1:57344 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260430 20:02:33.694561 18510 ts_tablet_manager.cc:1929] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:33.694653 18510 log.cc:1199] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/a6db5dee18e745058d104efe5642236f
I20260430 20:02:33.695176 18511 ts_tablet_manager.cc:1929] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:33.695271 18511 log.cc:1199] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/a6db5dee18e745058d104efe5642236f
I20260430 20:02:33.695271 18510 ts_tablet_manager.cc:1950] T a6db5dee18e745058d104efe5642236f P 63ee5bda423347388be745e1042e726d: Deleting consensus metadata
I20260430 20:02:33.695626 18511 ts_tablet_manager.cc:1950] T a6db5dee18e745058d104efe5642236f P 29adfe52affa45fc84188f9ae739e46f: Deleting consensus metadata
W20260430 20:02:33.696466 18342 connection.cc:463] server connection from 127.0.0.1:55710 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55710 (request call id 1) could send its response
W20260430 20:02:33.696678 18072 connection.cc:463] server connection from 127.0.0.1:60728 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60728 (request call id 1) could send its response
I20260430 20:02:33.706691 18051 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.697858 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.10.46.60:38489)
0430 20:02:33.697988 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.697993 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.698292 (+ 299us) server_negotiation.cc:408] Connection header received
0430 20:02:33.698442 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.698446 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.698503 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.698712 (+ 209us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.699859 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.700657 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.701391 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.701607 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.702892 (+ 1285us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.702913 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.702918 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.702947 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.704853 (+ 1906us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.705550 (+ 697us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.705552 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.705554 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.705600 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.705849 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.705853 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.705855 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.706021 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.706148 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.706371 (+ 223us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.706476 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 20:02:33.707268 18048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:33.697495 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.10.46.61:33315)
0430 20:02:33.697655 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:33.697662 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:33.698176 (+ 514us) server_negotiation.cc:408] Connection header received
0430 20:02:33.698305 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:33.698308 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:33.698365 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:33.698451 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:33.700250 (+ 1799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.700764 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:33.702009 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:33.702230 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:33.703350 (+ 1120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:33.703375 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:33.703379 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:33.703412 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:33.704955 (+ 1543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.705418 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.705421 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.705422 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.705465 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:33.705718 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:33.705721 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:33.705723 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:33.705900 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:33.705997 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:33.707073 (+ 1076us) server_negotiation.cc:300] Negotiation successful
0430 20:02:33.707160 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 20:02:34.669818 18520 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:34.669179 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:57727)
0430 20:02:34.669617 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:34.669702 (+ 85us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":101,"threads_started":1}
W20260430 20:02:34.670101 18455 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:37669 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:34.670267 18521 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:34.669444 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:37985)
0430 20:02:34.670097 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:34.670167 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":63,"threads_started":1}
W20260430 20:02:34.670656 18319 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:37669 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:34.671110 18522 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:34.670693 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:47889)
0430 20:02:34.670983 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:34.671035 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
W20260430 20:02:34.671442 18183 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:37669 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:35.181620 18523 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fd426cc2dce4b249bbd545a803b624b)
I20260430 20:02:35.181807 18523 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } }
I20260430 20:02:35.182075 18524 raft_consensus.cc:493] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fd426cc2dce4b249bbd545a803b624b)
I20260430 20:02:35.182278 18524 raft_consensus.cc:515] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } }
I20260430 20:02:35.182705 18523 leader_election.cc:290] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669), 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60:38489)
I20260430 20:02:35.183238 17999 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ec8f0e25ec749fca187e2322ac4004f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "77727b4f754f4d9aa38f310a91c2900d" is_pre_election: true
I20260430 20:02:35.183378 18525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.182838 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52270)
0430 20:02:35.183180 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.183260 (+ 80us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":60,"threads_started":1}
I20260430 20:02:35.183399 18524 leader_election.cc:290] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669), 7ec8f0e25ec749fca187e2322ac4004f (127.10.46.61:33315)
I20260430 20:02:35.183424 17999 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec8f0e25ec749fca187e2322ac4004f in term 1.
W20260430 20:02:35.183498 17790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:35.183951 17792 leader_election.cc:304] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77727b4f754f4d9aa38f310a91c2900d, 7ec8f0e25ec749fca187e2322ac4004f; no voters:
I20260430 20:02:35.184114 18527 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.183662 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52284)
0430 20:02:35.183925 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.183991 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":55,"threads_started":1}
I20260430 20:02:35.184190 18523 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 20:02:35.184255 18523 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fd426cc2dce4b249bbd545a803b624b)
I20260430 20:02:35.184309 18523 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 1 FOLLOWER]: Advancing to term 2
I20260430 20:02:35.184507 18525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.184082 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52286)
0430 20:02:35.184373 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.184419 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":68,"threads_started":1}
I20260430 20:02:35.184371 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77727b4f754f4d9aa38f310a91c2900d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ec8f0e25ec749fca187e2322ac4004f" is_pre_election: true
W20260430 20:02:35.184604 17790 leader_election.cc:336] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669): Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
W20260430 20:02:35.184909 17974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:35.185864 18527 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.185638 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52298)
0430 20:02:35.185769 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.185808 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260430 20:02:35.185832 18523 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } }
W20260430 20:02:35.186048 17974 leader_election.cc:336] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669): Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:35.186136 17815 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77727b4f754f4d9aa38f310a91c2900d in current term 2: Already voted for candidate 7ec8f0e25ec749fca187e2322ac4004f in this term.
I20260430 20:02:35.186590 18528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.186338 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52300)
0430 20:02:35.186465 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.186510 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260430 20:02:35.186647 18523 leader_election.cc:290] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 election: Requested vote from peers 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669), 77727b4f754f4d9aa38f310a91c2900d (127.10.46.60:38489)
I20260430 20:02:35.186777 17975 leader_election.cc:304] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77727b4f754f4d9aa38f310a91c2900d; no voters: 0fd426cc2dce4b249bbd545a803b624b, 7ec8f0e25ec749fca187e2322ac4004f
I20260430 20:02:35.186947 17999 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ec8f0e25ec749fca187e2322ac4004f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "77727b4f754f4d9aa38f310a91c2900d"
I20260430 20:02:35.187080 17999 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 1 FOLLOWER]: Advancing to term 2
I20260430 20:02:35.187294 18528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.187029 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52310)
0430 20:02:35.187164 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.187210 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260430 20:02:35.187517 17790 leader_election.cc:336] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669): Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
I20260430 20:02:35.188480 17999 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec8f0e25ec749fca187e2322ac4004f in term 2.
I20260430 20:02:35.188726 18524 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 20:02:35.189105 17792 leader_election.cc:304] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77727b4f754f4d9aa38f310a91c2900d, 7ec8f0e25ec749fca187e2322ac4004f; no voters: 0fd426cc2dce4b249bbd545a803b624b
I20260430 20:02:35.189364 18523 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 2 FOLLOWER]: Leader election won for term 2
I20260430 20:02:35.189839 18523 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [term 2 LEADER]: Becoming Leader. State: Replica: 7ec8f0e25ec749fca187e2322ac4004f, State: Running, Role: LEADER
I20260430 20:02:35.190201 18523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } }
I20260430 20:02:35.191679 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec8f0e25ec749fca187e2322ac4004f. Latest consensus state: current_term: 2 leader_uuid: "7ec8f0e25ec749fca187e2322ac4004f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:35.191897 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:35.192674 18532 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:35.196983 18532 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:35.197582 18532 catalog_manager.cc:1269] Loaded cluster ID: 378f4f0d68054f2184a1aeb75cd4f491
I20260430 20:02:35.197664 18532 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:35.198374 18532 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:35.198834 18532 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Loaded TSK: 0
I20260430 20:02:35.199469 18532 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:02:35.228471 17804 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:02:35.231014 17804 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:35.235363 17999 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [term 2 FOLLOWER]: Refusing update from remote peer 7ec8f0e25ec749fca187e2322ac4004f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260430 20:02:35.235527 18528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.234789 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52320)
0430 20:02:35.235119 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.235363 (+ 244us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260430 20:02:35.236195 18530 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260430 20:02:35.236920 18528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.236570 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.0.0.1:52324)
0430 20:02:35.236728 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.236811 (+ 83us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260430 20:02:35.237110 17790 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f -> Peer 0fd426cc2dce4b249bbd545a803b624b (127.10.46.62:37669): Couldn't send request to peer 0fd426cc2dce4b249bbd545a803b624b. Status: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260430 20:02:35.238306 18524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec8f0e25ec749fca187e2322ac4004f. Latest consensus state: current_term: 2 leader_uuid: "7ec8f0e25ec749fca187e2322ac4004f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:35.238485 18524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:35.239928 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7ec8f0e25ec749fca187e2322ac4004f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:35.240072 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:35.240444 18524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7ec8f0e25ec749fca187e2322ac4004f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0fd426cc2dce4b249bbd545a803b624b" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37669 } } peers { permanent_uuid: "7ec8f0e25ec749fca187e2322ac4004f" member_type: VOTER last_known_addr { host: "127.10.46.61" port: 33315 } attrs { promote: false } } peers { permanent_uuid: "77727b4f754f4d9aa38f310a91c2900d" member_type: VOTER last_known_addr { host: "127.10.46.60" port: 38489 } attrs { promote: false } } }
I20260430 20:02:35.240703 18524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77727b4f754f4d9aa38f310a91c2900d [sys.catalog]: This master's current role is: FOLLOWER
I20260430 20:02:35.240712 18534 mvcc.cc:204] Tried to move back new op lower bound from 7280965039035330560 to 7280965038861533184. Current Snapshot: MvccSnapshot[applied={T|T < 7280965039035330560}]
I20260430 20:02:35.254043 18528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.250422 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:43023 (local address 127.0.0.1:57352)
0430 20:02:35.250577 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.250595 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:35.250701 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:35.251517 (+ 816us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:35.251520 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:35.251536 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:35.251834 (+ 298us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.251841 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.252829 (+ 988us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.252833 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:35.253699 (+ 866us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.253707 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.253786 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.253803 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 20:02:35.253840 (+ 37us) client_negotiation.cc:241] Negotiation successful
0430 20:02:35.253905 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260430 20:02:35.254760 18525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.251086 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:45363 (local address 127.0.0.1:60730)
0430 20:02:35.251181 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.251200 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:35.251268 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:35.251741 (+ 473us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:35.251744 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:35.251759 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:35.251994 (+ 235us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.252000 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.253313 (+ 1313us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.253317 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:35.254263 (+ 946us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.254285 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.254481 (+ 196us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.254503 (+ 22us) client_negotiation.cc:770] Sending connection context
0430 20:02:35.254561 (+ 58us) client_negotiation.cc:241] Negotiation successful
0430 20:02:35.254630 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260430 20:02:35.255141 18539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.250706 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.10.46.2:43023)
0430 20:02:35.251037 (+ 331us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.251041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.251056 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:35.251357 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.251360 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.251407 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.251506 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:35.251985 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.252703 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.254363 (+ 1660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.254859 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.254896 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.254948 (+ 52us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.255007 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260430 20:02:35.255707 18541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.252099 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:34589 (local address 127.0.0.1:55732)
0430 20:02:35.252566 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.252579 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:35.252673 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:35.253190 (+ 517us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:35.253194 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:35.253213 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:35.253462 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.253468 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.254398 (+ 930us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.254402 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:35.255277 (+ 875us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.255288 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.255376 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.255389 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:35.255419 (+ 30us) client_negotiation.cc:241] Negotiation successful
0430 20:02:35.255581 (+ 162us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":118,"threads_started":1}
I20260430 20:02:35.255928 18252 tablet_service.cc:1511] Processing CreateTablet for tablet 13430a8e401348ea9f82eb8d2f70355e (DEFAULT_TABLE table=test-table [id=9f494b83829e498794edf02cfd411529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:35.256242 18252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13430a8e401348ea9f82eb8d2f70355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:35.256259 18540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.251217 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.10.46.1:45363)
0430 20:02:35.251493 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.251497 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.251512 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:35.251565 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.251568 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.251620 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.251724 (+ 104us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:35.252254 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.253216 (+ 962us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.255261 (+ 2045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.255819 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.256000 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.256055 (+ 55us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.256131 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260430 20:02:35.256764 18542 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.252462 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.10.46.3:34589)
0430 20:02:35.252933 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.252937 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.252953 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:02:35.253000 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.253002 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.253057 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.253180 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:35.253591 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.254255 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.255974 (+ 1719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.256462 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.256495 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.256553 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.256621 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":74,"threads_started":1}
I20260430 20:02:35.256772 18116 tablet_service.cc:1511] Processing CreateTablet for tablet 13430a8e401348ea9f82eb8d2f70355e (DEFAULT_TABLE table=test-table [id=9f494b83829e498794edf02cfd411529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:35.257057 18116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13430a8e401348ea9f82eb8d2f70355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:35.257218 18388 tablet_service.cc:1511] Processing CreateTablet for tablet 13430a8e401348ea9f82eb8d2f70355e (DEFAULT_TABLE table=test-table [id=9f494b83829e498794edf02cfd411529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:35.257524 18388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13430a8e401348ea9f82eb8d2f70355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:35.259003 18543 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: Bootstrap starting.
I20260430 20:02:35.259275 18544 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: Bootstrap starting.
I20260430 20:02:35.259900 18545 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Bootstrap starting.
I20260430 20:02:35.260246 18544 tablet_bootstrap.cc:654] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:35.260499 18543 tablet_bootstrap.cc:654] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:35.260888 18545 tablet_bootstrap.cc:654] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:35.261592 18544 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: No bootstrap required, opened a new log
I20260430 20:02:35.261699 18544 ts_tablet_manager.cc:1403] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:35.262109 18543 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: No bootstrap required, opened a new log
I20260430 20:02:35.262255 18543 ts_tablet_manager.cc:1403] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:35.262557 18545 tablet_bootstrap.cc:492] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: No bootstrap required, opened a new log
I20260430 20:02:35.262620 18544 raft_consensus.cc:359] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.262740 18545 ts_tablet_manager.cc:1403] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:35.262787 18544 raft_consensus.cc:385] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:35.262822 18544 raft_consensus.cc:740] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29adfe52affa45fc84188f9ae739e46f, State: Initialized, Role: FOLLOWER
I20260430 20:02:35.262789 18543 raft_consensus.cc:359] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.262909 18543 raft_consensus.cc:385] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:35.262952 18543 raft_consensus.cc:740] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1092265a9534ad5b673c04b278c87c3, State: Initialized, Role: FOLLOWER
I20260430 20:02:35.263016 18544 consensus_queue.cc:260] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.263109 18543 consensus_queue.cc:260] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.263311 18545 raft_consensus.cc:359] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.263451 18545 raft_consensus.cc:385] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:35.263494 18545 raft_consensus.cc:740] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee5bda423347388be745e1042e726d, State: Initialized, Role: FOLLOWER
I20260430 20:02:35.263506 18544 ts_tablet_manager.cc:1434] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:35.263521 18543 ts_tablet_manager.cc:1434] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:02:35.263724 18545 consensus_queue.cc:260] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.265273 18545 ts_tablet_manager.cc:1434] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:35.266590 18554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.266123 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:39739)
0430 20:02:35.266428 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.266488 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
I20260430 20:02:35.266994 18553 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.265531 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:37657)
0430 20:02:35.266830 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.266888 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":111,"threads_started":1}
I20260430 20:02:35.267184 18552 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.264500 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:53043)
0430 20:02:35.267036 (+ 2536us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.267091 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2441,"thread_start_us":119,"threads_started":1}
I20260430 20:02:35.268477 18552 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.268200 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:37995)
0430 20:02:35.268357 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.268404 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260430 20:02:35.268878 18456 heartbeater.cc:499] Master 127.10.46.61:33315 was elected leader, sending a full tablet report...
I20260430 20:02:35.269006 18320 heartbeater.cc:499] Master 127.10.46.61:33315 was elected leader, sending a full tablet report...
I20260430 20:02:35.269042 18554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.268200 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:36551)
0430 20:02:35.268919 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.268969 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653}
I20260430 20:02:35.271499 18553 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.271057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:60579)
0430 20:02:35.271309 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.271375 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260430 20:02:35.272034 18184 heartbeater.cc:499] Master 127.10.46.61:33315 was elected leader, sending a full tablet report...
I20260430 20:02:35.407089 18550 raft_consensus.cc:493] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:35.407249 18550 raft_consensus.cc:515] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.408665 18550 leader_election.cc:290] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363), 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:35.412252 18552 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.408939 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:34589 (local address 127.10.46.2:34989)
0430 20:02:35.409149 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.409165 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:35.409288 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:35.409719 (+ 431us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:35.409722 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:35.409751 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:35.410073 (+ 322us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.410085 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.411082 (+ 997us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.411086 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:35.411906 (+ 820us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.411915 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.412020 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.412036 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:35.412075 (+ 39us) client_negotiation.cc:241] Negotiation successful
0430 20:02:35.412129 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260430 20:02:35.413231 18555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.409070 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:45363 (local address 127.10.46.2:39975)
0430 20:02:35.409608 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.409626 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:35.409745 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:35.410098 (+ 353us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:35.410101 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:35.410134 (+ 33us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:35.410535 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.410543 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.411521 (+ 978us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.411525 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:35.412360 (+ 835us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:35.412376 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.412650 (+ 274us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.412665 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 20:02:35.412924 (+ 259us) client_negotiation.cc:241] Negotiation successful
0430 20:02:35.413028 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":201,"threads_started":1}
I20260430 20:02:35.413196 18542 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.409061 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:34989 (local address 127.10.46.3:34589)
0430 20:02:35.409195 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.409202 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.409219 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:35.409447 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.409451 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.409505 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.409546 (+ 41us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:35.410245 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.410954 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.412037 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.412649 (+ 612us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.412828 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.412922 (+ 94us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.413028 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 20:02:35.413232 18540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.409517 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:39975 (local address 127.10.46.1:45363)
0430 20:02:35.409731 (+ 214us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.409735 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.409753 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:02:35.409857 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.409860 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.409916 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.410012 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:35.410715 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.411349 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.412650 (+ 1301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.413008 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.413038 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.413088 (+ 50us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.413129 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260430 20:02:35.413815 18408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13430a8e401348ea9f82eb8d2f70355e" candidate_uuid: "a1092265a9534ad5b673c04b278c87c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee5bda423347388be745e1042e726d" is_pre_election: true
I20260430 20:02:35.413811 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13430a8e401348ea9f82eb8d2f70355e" candidate_uuid: "a1092265a9534ad5b673c04b278c87c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29adfe52affa45fc84188f9ae739e46f" is_pre_election: true
I20260430 20:02:35.413992 18136 raft_consensus.cc:2468] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1092265a9534ad5b673c04b278c87c3 in term 0.
I20260430 20:02:35.414005 18408 raft_consensus.cc:2468] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1092265a9534ad5b673c04b278c87c3 in term 0.
I20260430 20:02:35.414541 18206 leader_election.cc:304] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29adfe52affa45fc84188f9ae739e46f, 63ee5bda423347388be745e1042e726d, a1092265a9534ad5b673c04b278c87c3; no voters:
I20260430 20:02:35.414844 18550 raft_consensus.cc:2804] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:35.414917 18550 raft_consensus.cc:493] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:35.414958 18550 raft_consensus.cc:3060] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:35.416080 18550 raft_consensus.cc:515] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.416503 18550 leader_election.cc:290] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [CANDIDATE]: Term 1 election: Requested vote from peers 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363), 63ee5bda423347388be745e1042e726d (127.10.46.3:34589)
I20260430 20:02:35.417227 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13430a8e401348ea9f82eb8d2f70355e" candidate_uuid: "a1092265a9534ad5b673c04b278c87c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29adfe52affa45fc84188f9ae739e46f"
I20260430 20:02:35.417265 18408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13430a8e401348ea9f82eb8d2f70355e" candidate_uuid: "a1092265a9534ad5b673c04b278c87c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee5bda423347388be745e1042e726d"
I20260430 20:02:35.417425 18136 raft_consensus.cc:3060] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:35.417459 18408 raft_consensus.cc:3060] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:35.418520 18408 raft_consensus.cc:2468] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1092265a9534ad5b673c04b278c87c3 in term 1.
I20260430 20:02:35.419001 18206 leader_election.cc:304] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee5bda423347388be745e1042e726d, a1092265a9534ad5b673c04b278c87c3; no voters:
I20260430 20:02:35.419023 18136 raft_consensus.cc:2468] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1092265a9534ad5b673c04b278c87c3 in term 1.
I20260430 20:02:35.419315 18550 raft_consensus.cc:2804] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:35.419575 18550 raft_consensus.cc:697] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 1 LEADER]: Becoming Leader. State: Replica: a1092265a9534ad5b673c04b278c87c3, State: Running, Role: LEADER
I20260430 20:02:35.419915 18550 consensus_queue.cc:237] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } }
I20260430 20:02:35.420686 18555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.420372 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:33433)
0430 20:02:35.420553 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.420612 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260430 20:02:35.422936 17803 catalog_manager.cc:5671] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1092265a9534ad5b673c04b278c87c3 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "a1092265a9534ad5b673c04b278c87c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1092265a9534ad5b673c04b278c87c3" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 43023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:35.456995 18539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.453021 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.10.46.2:43023)
0430 20:02:35.453230 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:35.453234 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:35.453251 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:35.453433 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:35.453436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:35.453494 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:35.453619 (+ 125us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:35.454176 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.454933 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:35.455883 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:35.456057 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:35.456111 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:35.456517 (+ 406us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:35.456610 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:35.456748 (+ 138us) server_negotiation.cc:300] Negotiation successful
0430 20:02:35.456797 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260430 20:02:35.460249 18136 raft_consensus.cc:1275] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Refusing update from remote peer a1092265a9534ad5b673c04b278c87c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:35.460383 18408 raft_consensus.cc:1275] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Refusing update from remote peer a1092265a9534ad5b673c04b278c87c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:35.460988 18550 consensus_queue.cc:1048] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29adfe52affa45fc84188f9ae739e46f" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:35.461189 18556 consensus_queue.cc:1048] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee5bda423347388be745e1042e726d" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:35.463631 18553 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.463298 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:51785)
0430 20:02:35.463468 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.463536 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260430 20:02:35.463884 18554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.463543 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:48713)
0430 20:02:35.463720 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.463793 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260430 20:02:35.466902 18555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.466571 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:41321)
0430 20:02:35.466755 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.466807 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260430 20:02:35.467204 18553 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.466907 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.1:37579)
0430 20:02:35.467084 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.467130 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260430 20:02:35.468003 18555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.467720 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.2:49113)
0430 20:02:35.467890 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.467932 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260430 20:02:35.468894 18554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.468659 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37669 (local address 127.10.46.3:49727)
0430 20:02:35.468767 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.468820 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260430 20:02:35.470536 18561 mvcc.cc:204] Tried to move back new op lower bound from 7280965039958978560 to 7280965039803617280. Current Snapshot: MvccSnapshot[applied={T|T < 7280965039958978560}]
I20260430 20:02:35.479912 17805 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:35.480116 17805 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:35.484910 17805 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 7ec8f0e25ec749fca187e2322ac4004f: Sending DeleteTablet for 3 replicas of tablet 13430a8e401348ea9f82eb8d2f70355e
I20260430 20:02:35.485718 18252 tablet_service.cc:1558] Processing DeleteTablet for tablet 13430a8e401348ea9f82eb8d2f70355e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:35 UTC) from {username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:57352
I20260430 20:02:35.485710 18116 tablet_service.cc:1558] Processing DeleteTablet for tablet 13430a8e401348ea9f82eb8d2f70355e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:35 UTC) from {username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:60730
I20260430 20:02:35.486308 18569 tablet_replica.cc:333] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f: stopping tablet replica
I20260430 20:02:35.486294 18388 tablet_service.cc:1558] Processing DeleteTablet for tablet 13430a8e401348ea9f82eb8d2f70355e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:35 UTC) from {username='oryx', principal='oryx/127.10.46.61@KRBTEST.COM'} at 127.0.0.1:55732
I20260430 20:02:35.486519 18569 raft_consensus.cc:2243] T 13430a8e401348ea9f82eb8d2f70355e P 29adfe52affa45fc84188f9ae739e46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:35.486675 18571 tablet_replica.cc:333] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: stopping tablet replica
I20260430 20:02:35.486788 18571 raft_consensus.cc:2243] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:35.486932 18571 raft_consensus.cc:2272] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:35.487330 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18053
I20260430 20:02:35.488085 18571 ts_tablet_manager.cc:1916] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:35.489960 18571 ts_tablet_manager.cc:1929] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:35.490065 18571 log.cc:1199] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/13430a8e401348ea9f82eb8d2f70355e
I20260430 20:02:35.490343 18571 ts_tablet_manager.cc:1950] T 13430a8e401348ea9f82eb8d2f70355e P 63ee5bda423347388be745e1042e726d: Deleting consensus metadata
I20260430 20:02:35.491117 17790 catalog_manager.cc:5002] TS 63ee5bda423347388be745e1042e726d (127.10.46.3:34589): tablet 13430a8e401348ea9f82eb8d2f70355e (table test-table [id=9f494b83829e498794edf02cfd411529]) successfully deleted
I20260430 20:02:35.495697 18570 tablet_replica.cc:333] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: stopping tablet replica
I20260430 20:02:35.495954 18570 raft_consensus.cc:2243] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:35.496345 18570 raft_consensus.cc:2272] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:35.497257 18570 ts_tablet_manager.cc:1916] T 13430a8e401348ea9f82eb8d2f70355e P a1092265a9534ad5b673c04b278c87c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:35.498409 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18189
I20260430 20:02:35.498493 18541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.498164 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:45363 (local address 127.0.0.1:33166)
0430 20:02:35.498330 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.498393 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260430 20:02:35.500072 17792 catalog_manager.cc:4729] TS 29adfe52affa45fc84188f9ae739e46f (127.10.46.1:45363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13430a8e401348ea9f82eb8d2f70355e: Network error: Client connection negotiation failed: client connection to 127.10.46.1:45363: connect: Connection refused (error 111)
I20260430 20:02:35.505554 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18325
I20260430 20:02:35.505887 18541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:35.505528 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:43023 (local address 127.0.0.1:36206)
0430 20:02:35.505713 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:35.505772 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.2:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260430 20:02:35.512872 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17666
I20260430 20:02:35.519418 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 17777
2026-04-30T20:02:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7931 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:35 dist-test-slave-0bdw krb5kdc[18574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:35 dist-test-slave-0bdw krb5kdc[18574](info): set up 2 sockets
Apr 30 20:02:35 dist-test-slave-0bdw krb5kdc[18574](info): commencing operation
krb5kdc: starting...
W20260430 20:02:37.657533 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:37 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:37.825194 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35705
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:37.954945 18590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:37.955295 18590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:37.955417 18590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:37.959969 18590 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:37.960091 18590 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 20:02:37.960126 18590 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:37.960181 18590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:37.960246 18590 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:37.960283 18590 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:37.966619 18590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:35705
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:37.968241 18590 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:37.969187 18590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:37.975715 18598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:37.975831 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:37.975715 18596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:37.976274 18590 server_base.cc:1061] running on GCE node
I20260430 20:02:37.977587 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:37.979067 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:37.980351 18590 hybrid_clock.cc:648] HybridClock initialized: now 1777579357980313 us; error 57 us; skew 500 ppm
Apr 30 20:02:37 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579357, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:37.984854 18590 init.cc:377] Logged in from keytab as oryx/127.10.46.62@KRBTEST.COM (short username oryx)
I20260430 20:02:37.986442 18590 webserver.cc:492] Webserver started at http://127.10.46.62:45159/ using document root <none> and password file <none>
I20260430 20:02:37.987102 18590 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:37.987198 18590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:37.987457 18590 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:37.989435 18590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "02deb7bf8cf6468c81da741a98bf5297"
format_stamp: "Formatted at 2026-04-30 20:02:37 on dist-test-slave-0bdw"
server_key: "647dfafdfddcd33da92be412be5b9583"
server_key_iv: "7661a4a4a305ddb8c53fd889fdf1fe8d"
server_key_version: "encryptionkey@0"
I20260430 20:02:37.989984 18590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "02deb7bf8cf6468c81da741a98bf5297"
format_stamp: "Formatted at 2026-04-30 20:02:37 on dist-test-slave-0bdw"
server_key: "647dfafdfddcd33da92be412be5b9583"
server_key_iv: "7661a4a4a305ddb8c53fd889fdf1fe8d"
server_key_version: "encryptionkey@0"
I20260430 20:02:37.994441 18590 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260430 20:02:37.997119 18605 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:37.998939 18590 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260430 20:02:37.999126 18590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "02deb7bf8cf6468c81da741a98bf5297"
format_stamp: "Formatted at 2026-04-30 20:02:37 on dist-test-slave-0bdw"
server_key: "647dfafdfddcd33da92be412be5b9583"
server_key_iv: "7661a4a4a305ddb8c53fd889fdf1fe8d"
server_key_version: "encryptionkey@0"
I20260430 20:02:37.999270 18590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:38.029172 18590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:38.032675 18590 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:38.032922 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:38.046089 18590 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:35705
I20260430 20:02:38.046406 18667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:35705 every 8 connection(s)
I20260430 20:02:38.047260 18590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:38.050837 18668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:38.053617 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18590
I20260430 20:02:38.053787 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:38.054044 10424 external_mini_cluster.cc:1468] Setting key 4e57d0d7d7f6f9178301ce389471bfa9
I20260430 20:02:38.056677 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Bootstrap starting.
I20260430 20:02:38.059813 18668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:38.060992 18668 log.cc:826] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:38.063853 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: No bootstrap required, opened a new log
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:38.066823 18668 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } }
I20260430 20:02:38.067073 18668 raft_consensus.cc:385] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:38.067173 18668 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02deb7bf8cf6468c81da741a98bf5297, State: Initialized, Role: FOLLOWER
I20260430 20:02:38.067626 18668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } }
I20260430 20:02:38.067778 18668 raft_consensus.cc:399] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:38.067895 18668 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:38.068010 18668 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:38.068980 18668 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } }
I20260430 20:02:38.069376 18668 leader_election.cc:304] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02deb7bf8cf6468c81da741a98bf5297; no voters:
I20260430 20:02:38.069749 18668 leader_election.cc:290] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:38.069835 18673 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:38.069998 18673 raft_consensus.cc:697] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [term 1 LEADER]: Becoming Leader. State: Replica: 02deb7bf8cf6468c81da741a98bf5297, State: Running, Role: LEADER
I20260430 20:02:38.070292 18673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } }
I20260430 20:02:38.071029 18668 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:38.071671 18675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02deb7bf8cf6468c81da741a98bf5297. Latest consensus state: current_term: 1 leader_uuid: "02deb7bf8cf6468c81da741a98bf5297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } } }
I20260430 20:02:38.071857 18675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:38.071611 18674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02deb7bf8cf6468c81da741a98bf5297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02deb7bf8cf6468c81da741a98bf5297" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 35705 } } }
I20260430 20:02:38.072031 18674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:38.074194 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.055687 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.10.46.62:35705)
0430 20:02:38.056555 (+ 868us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.056565 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.056601 (+ 36us) server_negotiation.cc:408] Connection header received
0430 20:02:38.057328 (+ 727us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.057350 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.057989 (+ 639us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.058479 (+ 490us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.060205 (+ 1726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.061565 (+ 1360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.062680 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.063030 (+ 350us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.065986 (+ 2956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.066013 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.066029 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.066066 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.069199 (+ 3133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.069777 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.069783 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.069788 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.069919 (+ 131us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.070706 (+ 787us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.070710 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.070729 (+ 19us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.071420 (+ 691us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.072578 (+ 1158us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.072841 (+ 263us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.073328 (+ 487us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":234,"threads_started":1}
W20260430 20:02:38.076922 18688 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:38.077018 18688 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:38.077090 18678 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:38.077831 18678 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:38.085893 18678 catalog_manager.cc:1357] Generated new cluster ID: 294addffe20d4926827450840dcab59c
I20260430 20:02:38.085994 18678 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:38.101150 18678 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:38.103510 18678 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:38.113356 18678 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Generated new TSK 0
I20260430 20:02:38.114755 18678 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:38.188205 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35705
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:38.300611 18696 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:38.300853 18696 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:38.300915 18696 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:38.304577 18696 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:38.304656 18696 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:38.304711 18696 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:02:38.304756 18696 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:38.310170 18696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:38.311530 18696 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:38.312448 18696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:38.319617 18702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.319622 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.319617 18701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:38.320030 18696 server_base.cc:1061] running on GCE node
I20260430 20:02:38.320441 18696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:38.321033 18696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:38.322218 18696 hybrid_clock.cc:648] HybridClock initialized: now 1777579358322177 us; error 56 us; skew 500 ppm
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:38.325183 18696 init.cc:377] Logged in from keytab as oryx/127.10.46.1@KRBTEST.COM (short username oryx)
I20260430 20:02:38.326478 18696 webserver.cc:492] Webserver started at http://127.10.46.1:38081/ using document root <none> and password file <none>
I20260430 20:02:38.327075 18696 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:38.327126 18696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:38.327349 18696 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:38.329245 18696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "56aecaba07891e67bbfaca6819b34b83"
server_key_iv: "c1670edb253b5db24fac59ff5ca25052"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.329773 18696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "56aecaba07891e67bbfaca6819b34b83"
server_key_iv: "c1670edb253b5db24fac59ff5ca25052"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.333600 18696 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260430 20:02:38.335995 18711 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.337097 18696 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 20:02:38.337203 18696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "56aecaba07891e67bbfaca6819b34b83"
server_key_iv: "c1670edb253b5db24fac59ff5ca25052"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.337369 18696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:38.356354 18696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:38.360148 18696 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:38.360332 18696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:38.361938 18696 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:38.362039 18696 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.362102 18696 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:38.362161 18696 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:38.375224 18696 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:38923
I20260430 20:02:38.376566 18696 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:38.377753 18826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:38923 every 8 connection(s)
I20260430 20:02:38.378539 18720 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.364057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.1:55291)
0430 20:02:38.364752 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.364804 (+ 52us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.365776 (+ 972us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.366368 (+ 592us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.366377 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.366979 (+ 602us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.367697 (+ 718us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.367716 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.368611 (+ 895us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.368619 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.369295 (+ 676us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.369307 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.369589 (+ 282us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.370383 (+ 794us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.370414 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.372460 (+ 2046us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.375125 (+ 2665us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.375137 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.375153 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.375545 (+ 392us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.376105 (+ 560us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.376110 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.376112 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.376240 (+ 128us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.376890 (+ 650us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.376899 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.377348 (+ 449us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.377568 (+ 220us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.377873 (+ 305us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":160,"threads_started":1}
I20260430 20:02:38.379159 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.363670 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:55291 (local address 127.10.46.62:35705)
0430 20:02:38.363857 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.363861 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.364912 (+ 1051us) server_negotiation.cc:408] Connection header received
0430 20:02:38.366004 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.366008 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.366075 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.366190 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.367896 (+ 1706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.368431 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.369451 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.369690 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.372766 (+ 3076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.372798 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.372802 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.372840 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.374943 (+ 2103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.375858 (+ 915us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.375864 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.375866 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.375962 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.376483 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.376487 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.376489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.376756 (+ 267us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.376863 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.378907 (+ 2044us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.379043 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 20:02:38.381470 18827 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35705
I20260430 20:02:38.381734 18827 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:38.382349 18827 heartbeater.cc:507] Master 127.10.46.62:35705 requested a full tablet report, sending...
I20260430 20:02:38.384141 18621 ts_manager.cc:194] Registered new tserver with Master: 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923)
I20260430 20:02:38.385207 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18696
I20260430 20:02:38.385380 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:38.385821 10424 external_mini_cluster.cc:1468] Setting key 7c84e0902da3344d91d0e042339961a9
I20260430 20:02:38.385854 18621 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:55291
I20260430 20:02:38.396222 18833 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.386293 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.1:41133)
0430 20:02:38.386962 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.386981 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.387128 (+ 147us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.387415 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.387418 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.387858 (+ 440us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.388121 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.388128 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.388936 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.388941 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.389329 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.389335 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.389471 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.390148 (+ 677us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.390162 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.390696 (+ 534us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.393222 (+ 2526us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.393227 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.393229 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.393702 (+ 473us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.394549 (+ 847us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.394553 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.394555 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.394652 (+ 97us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.395403 (+ 751us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.395407 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.395542 (+ 135us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.395681 (+ 139us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.395973 (+ 292us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":137,"threads_started":1}
I20260430 20:02:38.396201 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.385027 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:41133 (local address 127.10.46.62:35705)
0430 20:02:38.385204 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.385209 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.387068 (+ 1859us) server_negotiation.cc:408] Connection header received
0430 20:02:38.387246 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.387249 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.387299 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.387391 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.388250 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.388815 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.389451 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.389652 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.390944 (+ 1292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.390963 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.390966 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.390990 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.392856 (+ 1866us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.393928 (+ 1072us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.393933 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.393935 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.394258 (+ 323us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.394885 (+ 627us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.394890 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.394892 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.395128 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.395408 (+ 280us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.395763 (+ 355us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.395974 (+ 211us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for oryx/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:38.461998 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35705
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:38.582391 18838 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:38.582721 18838 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:38.582845 18838 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:38.586791 18838 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:38.586895 18838 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:38.586953 18838 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:02:38.586997 18838 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:38.591861 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:38.593210 18838 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:38.594377 18838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:38.602061 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.602149 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.602070 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:38.602742 18838 server_base.cc:1061] running on GCE node
I20260430 20:02:38.603291 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:38.604003 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:38.605206 18838 hybrid_clock.cc:648] HybridClock initialized: now 1777579358605149 us; error 76 us; skew 500 ppm
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:38.608606 18838 init.cc:377] Logged in from keytab as oryx/127.10.46.2@KRBTEST.COM (short username oryx)
I20260430 20:02:38.610560 18838 webserver.cc:492] Webserver started at http://127.10.46.2:36573/ using document root <none> and password file <none>
I20260430 20:02:38.611415 18838 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:38.611624 18838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:38.612247 18838 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:38.615402 18838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "8172e68f2b984b06ae18a352a74ed834"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "680b5e816063ed0792fe49e3c95c8b55"
server_key_iv: "a51921a9b22aaf11d4337a9b8689e22f"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.616258 18838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "8172e68f2b984b06ae18a352a74ed834"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "680b5e816063ed0792fe49e3c95c8b55"
server_key_iv: "a51921a9b22aaf11d4337a9b8689e22f"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.620807 18838 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260430 20:02:38.624181 18853 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.625779 18838 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260430 20:02:38.625942 18838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "8172e68f2b984b06ae18a352a74ed834"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "680b5e816063ed0792fe49e3c95c8b55"
server_key_iv: "a51921a9b22aaf11d4337a9b8689e22f"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.626048 18838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:38.644660 18838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:38.648211 18838 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:38.648399 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:38.649988 18838 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:38.650051 18838 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.650094 18838 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:38.650136 18838 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:38.669481 18838 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:41743
I20260430 20:02:38.669516 18968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:41743 every 8 connection(s)
I20260430 20:02:38.670819 18838 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:38.672328 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.651032 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:34717 (local address 127.10.46.62:35705)
0430 20:02:38.651192 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.651196 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.651985 (+ 789us) server_negotiation.cc:408] Connection header received
0430 20:02:38.653640 (+ 1655us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.653644 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.653715 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.653815 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.657025 (+ 3210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.657647 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.660128 (+ 2481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.660370 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.667574 (+ 7204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.667600 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.667603 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.667638 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.669460 (+ 1822us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.670439 (+ 979us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.670442 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.670444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.670497 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.670937 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.670944 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.670948 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.671145 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.671304 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.672014 (+ 710us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.672162 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 20:02:38.673686 18862 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.651177 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.2:34717)
0430 20:02:38.651768 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.651880 (+ 112us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.653370 (+ 1490us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.654070 (+ 700us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.654085 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.654758 (+ 673us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.656665 (+ 1907us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.656709 (+ 44us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.658322 (+ 1613us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.658330 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.659897 (+ 1567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.659918 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.660484 (+ 566us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.663294 (+ 2810us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.663328 (+ 34us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.667307 (+ 3979us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.669916 (+ 2609us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.669923 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.669935 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.670267 (+ 332us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.670618 (+ 351us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.670622 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.670624 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.670748 (+ 124us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.671292 (+ 544us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.671301 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.671633 (+ 332us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.671918 (+ 285us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.672445 (+ 527us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":164,"threads_started":1}
I20260430 20:02:38.676322 18969 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35705
I20260430 20:02:38.676653 18969 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:38.677224 18969 heartbeater.cc:507] Master 127.10.46.62:35705 requested a full tablet report, sending...
I20260430 20:02:38.680135 18622 ts_manager.cc:194] Registered new tserver with Master: 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743)
I20260430 20:02:38.680397 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18838
I20260430 20:02:38.680498 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:02:38.680776 10424 external_mini_cluster.cc:1468] Setting key 422174ab4a49c72db8d463c9e376a17f
I20260430 20:02:38.680927 18622 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:34717
I20260430 20:02:38.689327 18975 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.678115 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.2:55293)
0430 20:02:38.678416 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.678430 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.678520 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.679419 (+ 899us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.679500 (+ 81us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.680341 (+ 841us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.680896 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.680903 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.682531 (+ 1628us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.682538 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.683099 (+ 561us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.683109 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.683247 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.684179 (+ 932us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.684194 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.684543 (+ 349us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.687212 (+ 2669us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.687285 (+ 73us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.687288 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.687592 (+ 304us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.688205 (+ 613us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.688213 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.688216 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.688294 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.688725 (+ 431us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.688728 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.688835 (+ 107us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.688951 (+ 116us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.689126 (+ 175us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260430 20:02:38.689308 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.678374 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:55293 (local address 127.10.46.62:35705)
0430 20:02:38.678943 (+ 569us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.678947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.678959 (+ 12us) server_negotiation.cc:408] Connection header received
0430 20:02:38.679121 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.679127 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.679184 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.679283 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.681143 (+ 1860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.682337 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.683226 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.683435 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.684725 (+ 1290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.684751 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.684754 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.684779 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.686995 (+ 2216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.687951 (+ 956us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.687956 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.687958 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.688039 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.688407 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.688411 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.688413 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.688614 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.688738 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.689005 (+ 267us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.689126 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
WARNING: no policy specified for oryx/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab.
I20260430 20:02:38.751978 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:35705
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:38.871519 18980 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:38.872080 18980 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:38.872298 18980 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:38.876209 18980 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:38.876335 18980 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:38.876425 18980 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 20:02:38.876521 18980 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:38.882433 18980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/krb5kdc/oryx.keytab
--principal=oryx/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.46.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:38.884222 18980 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:38.885699 18980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:38.894784 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.894784 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:38.894853 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:38.895677 18980 server_base.cc:1061] running on GCE node
I20260430 20:02:38.896191 18980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:38.896884 18980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:38.898104 18980 hybrid_clock.cc:648] HybridClock initialized: now 1777579358898078 us; error 50 us; skew 500 ppm
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:38.902101 18980 init.cc:377] Logged in from keytab as oryx/127.10.46.3@KRBTEST.COM (short username oryx)
I20260430 20:02:38.903661 18980 webserver.cc:492] Webserver started at http://127.10.46.3:34545/ using document root <none> and password file <none>
I20260430 20:02:38.904412 18980 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:38.904495 18980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:38.904732 18980 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:38.906615 18980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "1c32797c40f74a11be42c0824898e583"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "09987ffce38a2781ff95e55d2c55c508"
server_key_iv: "026af68a87dc29c48d20f341b9a9d64e"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.907142 18980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "1c32797c40f74a11be42c0824898e583"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "09987ffce38a2781ff95e55d2c55c508"
server_key_iv: "026af68a87dc29c48d20f341b9a9d64e"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.910864 18980 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 20:02:38.913448 18995 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.914683 18980 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 20:02:38.914855 18980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "1c32797c40f74a11be42c0824898e583"
format_stamp: "Formatted at 2026-04-30 20:02:38 on dist-test-slave-0bdw"
server_key: "09987ffce38a2781ff95e55d2c55c508"
server_key_iv: "026af68a87dc29c48d20f341b9a9d64e"
server_key_version: "encryptionkey@0"
I20260430 20:02:38.914988 18980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:38.931447 18980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:38.935318 18980 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:38.935571 18980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:38.937562 18980 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:38.937650 18980 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:38.937719 18980 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:38.937784 18980 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:38.951004 18980 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:46787
I20260430 20:02:38.951027 19110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:46787 every 8 connection(s)
I20260430 20:02:38.952289 18980 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:38.952641 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.938265 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:45065 (local address 127.10.46.62:35705)
0430 20:02:38.938443 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.938449 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.939185 (+ 736us) server_negotiation.cc:408] Connection header received
0430 20:02:38.940700 (+ 1515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.940708 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.940804 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.940873 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.943316 (+ 2443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.944249 (+ 933us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.945313 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.945501 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.948312 (+ 2811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.948336 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.948339 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.948370 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.950280 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.950993 (+ 713us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.950998 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.951001 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.951062 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.951442 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.951447 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.951449 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.951636 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.951759 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.952320 (+ 561us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.952452 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260430 20:02:38.953416 19004 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.938576 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.3:45065)
0430 20:02:38.939024 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.939065 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.940223 (+ 1158us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.941500 (+ 1277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.941517 (+ 17us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.942376 (+ 859us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.943080 (+ 704us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.943099 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.944577 (+ 1478us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.944583 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.945140 (+ 557us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.945149 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.945429 (+ 280us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.946188 (+ 759us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.946209 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.948105 (+ 1896us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.950445 (+ 2340us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.950452 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.950466 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.950841 (+ 375us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.951183 (+ 342us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.951188 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.951193 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.951323 (+ 130us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.951756 (+ 433us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.951763 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.952041 (+ 278us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.952262 (+ 221us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.952542 (+ 280us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260430 20:02:38.954958 19111 heartbeater.cc:344] Connected to a master server at 127.10.46.62:35705
I20260430 20:02:38.955163 19111 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:38.956074 19111 heartbeater.cc:507] Master 127.10.46.62:35705 requested a full tablet report, sending...
I20260430 20:02:38.957563 18621 ts_manager.cc:194] Registered new tserver with Master: 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:38.958542 18621 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:45065
I20260430 20:02:38.961387 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 18980
I20260430 20:02:38.961541 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:38.961900 10424 external_mini_cluster.cc:1468] Setting key 23b255d6c9a00dabd5bfcf77067fef22
I20260430 20:02:38.964864 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:02:38.966341 19117 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.957690 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.10.46.3:50167)
0430 20:02:38.958236 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:38.958261 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:38.958425 (+ 164us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:38.958850 (+ 425us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:38.958853 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:38.959100 (+ 247us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:38.959535 (+ 435us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.959550 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.960482 (+ 932us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.960486 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:38.961057 (+ 571us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:38.961067 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.961208 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.961841 (+ 633us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:38.961856 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:38.962227 (+ 371us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:38.964771 (+ 2544us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.964778 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.964783 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.965018 (+ 235us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.965347 (+ 329us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:38.965350 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:38.965351 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:38.965406 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:38.965872 (+ 466us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:38.965875 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:38.965962 (+ 87us) client_negotiation.cc:770] Sending connection context
0430 20:02:38.966054 (+ 92us) client_negotiation.cc:241] Negotiation successful
0430 20:02:38.966185 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":172,"threads_started":1}
I20260430 20:02:38.966509 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.957789 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:50167 (local address 127.10.46.62:35705)
0430 20:02:38.958266 (+ 477us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.958270 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.958403 (+ 133us) server_negotiation.cc:408] Connection header received
0430 20:02:38.958542 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.958545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.958606 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.958775 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.959737 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.960325 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.961213 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.961434 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.962598 (+ 1164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.962610 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.962612 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.962636 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.964638 (+ 2002us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.965140 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.965144 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.965146 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.965200 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:38.965508 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:38.965511 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:38.965513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:38.965749 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:38.965866 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:38.966143 (+ 277us) server_negotiation.cc:300] Negotiation successful
0430 20:02:38.966275 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
W20260430 20:02:38.990837 18671 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:38.983977 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.10.46.62:35705)
0430 20:02:38.984181 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.984190 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.984203 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:38.984246 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.984249 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.984296 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.984394 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.985287 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.985934 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.986678 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.986884 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.990687 (+ 3803us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53592: BlockingRecv error: recv got EOF from 127.0.0.1:53592 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Apr 30 20:02:38 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:39.002170 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:38.992610 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.10.46.62:35705)
0430 20:02:38.992774 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:38.992778 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:38.992957 (+ 179us) server_negotiation.cc:408] Connection header received
0430 20:02:38.993115 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:38.993118 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:38.993167 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:38.993279 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:38.994095 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.994720 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:38.995557 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:38.995855 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:38.998106 (+ 2251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:38.998130 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:38.998133 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:38.998166 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:38.999854 (+ 1688us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.000368 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.000373 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.000375 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.000594 (+ 219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.001173 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.001177 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.001179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.001392 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:39.001544 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.001847 (+ 303us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.002000 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260430 20:02:39.006181 18621 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:02:39.008880 18621 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:39.024797 19133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.020076 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:46787 (local address 127.0.0.1:42416)
0430 20:02:39.020695 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.020732 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.020872 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.021623 (+ 751us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.021630 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.021655 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.021971 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.021980 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.023312 (+ 1332us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.023315 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.024251 (+ 936us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.024265 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.024397 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.024458 (+ 61us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.024555 (+ 97us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.024632 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":98,"threads_started":1}
I20260430 20:02:39.025115 19136 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.021105 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.0.0.1:33468)
0430 20:02:39.021709 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.021724 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.021805 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.022194 (+ 389us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.022199 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.022222 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.022511 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.022518 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.023837 (+ 1319us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.023841 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.024742 (+ 901us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.024751 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.024855 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.024869 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.024915 (+ 46us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.024976 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":83,"threads_started":1}
I20260430 20:02:39.025383 19134 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.020198 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.10.46.3:46787)
0430 20:02:39.020748 (+ 550us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.020757 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.020785 (+ 28us) server_negotiation.cc:408] Connection header received
0430 20:02:39.021063 (+ 278us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.021069 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.021258 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.021421 (+ 163us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.022126 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.023185 (+ 1059us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.024408 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.024941 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.025063 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.025119 (+ 56us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.025244 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":145,"threads_started":1}
I20260430 20:02:39.025890 19135 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.020749 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:41743 (local address 127.0.0.1:33742)
0430 20:02:39.021035 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.021049 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.021136 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.022326 (+ 1190us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.022329 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.022344 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.022564 (+ 220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.022572 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.024191 (+ 1619us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.024195 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.025527 (+ 1332us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.025537 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.025652 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.025665 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.025712 (+ 47us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.025765 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260430 20:02:39.026522 19138 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.021495 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.10.46.1:38923)
0430 20:02:39.021823 (+ 328us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.021831 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.021853 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:02:39.021932 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.021936 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.022035 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.022194 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.022643 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.023688 (+ 1045us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.025361 (+ 1673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.025916 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.026024 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.026106 (+ 82us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.026381 (+ 275us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":91,"threads_started":1}
I20260430 20:02:39.027539 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.021088 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.10.46.2:41743)
0430 20:02:39.021747 (+ 659us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.021754 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.021775 (+ 21us) server_negotiation.cc:408] Connection header received
0430 20:02:39.021847 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.021852 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.021986 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.022162 (+ 176us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.022687 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.024073 (+ 1386us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.026141 (+ 2068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.027248 (+ 1107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.027320 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.027373 (+ 53us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.027439 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":100,"threads_started":1}
I20260430 20:02:39.028510 19045 tablet_service.cc:1511] Processing CreateTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (DEFAULT_TABLE table=test-table [id=19a6937e0fd24d3c92128e02cb08b80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.029047 18761 tablet_service.cc:1511] Processing CreateTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (DEFAULT_TABLE table=test-table [id=19a6937e0fd24d3c92128e02cb08b80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.029574 19045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab438d0a11241908a7ee0b1bd85e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.029878 18761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab438d0a11241908a7ee0b1bd85e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.029652 18903 tablet_service.cc:1511] Processing CreateTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (DEFAULT_TABLE table=test-table [id=19a6937e0fd24d3c92128e02cb08b80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.030447 18903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab438d0a11241908a7ee0b1bd85e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.036720 19141 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Bootstrap starting.
I20260430 20:02:39.037556 19140 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Bootstrap starting.
I20260430 20:02:39.039247 19141 tablet_bootstrap.cc:654] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.040171 19140 tablet_bootstrap.cc:654] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.040537 19141 log.cc:826] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:39.040805 19139 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Bootstrap starting.
I20260430 20:02:39.041662 19140 log.cc:826] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:39.042675 19141 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: No bootstrap required, opened a new log
I20260430 20:02:39.042897 19141 ts_tablet_manager.cc:1403] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260430 20:02:39.043617 19139 tablet_bootstrap.cc:654] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.043673 19140 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: No bootstrap required, opened a new log
I20260430 20:02:39.043891 19140 ts_tablet_manager.cc:1403] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260430 20:02:39.044822 19139 log.cc:826] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:39.046036 19141 raft_consensus.cc:359] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.046258 19141 raft_consensus.cc:385] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.046309 19141 raft_consensus.cc:740] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.046737 19141 consensus_queue.cc:260] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.046890 19139 tablet_bootstrap.cc:492] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: No bootstrap required, opened a new log
I20260430 20:02:39.047081 19139 ts_tablet_manager.cc:1403] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260430 20:02:39.047013 19140 raft_consensus.cc:359] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.047197 19140 raft_consensus.cc:385] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.047256 19140 raft_consensus.cc:740] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.047621 19141 ts_tablet_manager.cc:1434] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:02:39.047696 19140 consensus_queue.cc:260] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.047981 18969 heartbeater.cc:499] Master 127.10.46.62:35705 was elected leader, sending a full tablet report...
I20260430 20:02:39.048552 19140 ts_tablet_manager.cc:1434] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260430 20:02:39.048869 19111 heartbeater.cc:499] Master 127.10.46.62:35705 was elected leader, sending a full tablet report...
I20260430 20:02:39.050762 19147 raft_consensus.cc:493] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:39.050974 19147 raft_consensus.cc:515] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.051170 19139 raft_consensus.cc:359] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.051407 19139 raft_consensus.cc:385] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.051489 19139 raft_consensus.cc:740] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.052006 19139 consensus_queue.cc:260] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.052407 19147 leader_election.cc:290] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787), 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923)
I20260430 20:02:39.053403 19139 ts_tablet_manager.cc:1434] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260430 20:02:39.053786 18827 heartbeater.cc:499] Master 127.10.46.62:35705 was elected leader, sending a full tablet report...
I20260430 20:02:39.056095 18862 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.052597 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:46787 (local address 127.10.46.2:55137)
0430 20:02:39.052727 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.052745 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.052849 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.053376 (+ 527us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.053379 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.053399 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.053666 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.053674 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.054745 (+ 1071us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.054749 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.055682 (+ 933us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.055691 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.055825 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.055846 (+ 21us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.055903 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.055966 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260430 20:02:39.057497 19134 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.052997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:55137 (local address 127.10.46.3:46787)
0430 20:02:39.053123 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.053127 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.053142 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:39.053190 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.053193 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.053243 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.053362 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.053801 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.054615 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.056368 (+ 1753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.057166 (+ 798us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.057225 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.057283 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.057357 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260430 20:02:39.058328 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ab438d0a11241908a7ee0b1bd85e7b2" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583" is_pre_election: true
I20260430 20:02:39.058614 19065 raft_consensus.cc:2468] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.059156 18854 leader_election.cc:304] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32797c40f74a11be42c0824898e583, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.059496 19145 raft_consensus.cc:2804] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:39.059587 19145 raft_consensus.cc:493] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:39.059633 19145 raft_consensus.cc:3060] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.060626 19149 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.052867 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.10.46.2:51397)
0430 20:02:39.056509 (+ 3642us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.056539 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.056887 (+ 348us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.057272 (+ 385us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.057275 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.057294 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.057577 (+ 283us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.057599 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.058878 (+ 1279us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.058885 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.059924 (+ 1039us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.059932 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.060068 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.060086 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.060389 (+ 303us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.060458 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2745,"thread_start_us":124,"threads_started":1}
I20260430 20:02:39.060894 19145 raft_consensus.cc:515] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.061110 19138 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.053155 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:51397 (local address 127.10.46.1:38923)
0430 20:02:39.053276 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.053280 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.056799 (+ 3519us) server_negotiation.cc:408] Connection header received
0430 20:02:39.057018 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.057022 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.057076 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.057132 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.057732 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.058751 (+ 1019us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.060049 (+ 1298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.060827 (+ 778us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.060870 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.060924 (+ 54us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.060981 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260430 20:02:39.061662 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ab438d0a11241908a7ee0b1bd85e7b2" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" is_pre_election: true
I20260430 20:02:39.061923 18781 raft_consensus.cc:2468] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.062588 19145 leader_election.cc:290] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Requested vote from peers 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787), 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923)
I20260430 20:02:39.063055 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ab438d0a11241908a7ee0b1bd85e7b2" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583"
I20260430 20:02:39.063064 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ab438d0a11241908a7ee0b1bd85e7b2" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
I20260430 20:02:39.063189 18781 raft_consensus.cc:3060] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.063201 19065 raft_consensus.cc:3060] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.064355 19065 raft_consensus.cc:2468] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:39.064447 18781 raft_consensus.cc:2468] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:39.064913 18854 leader_election.cc:304] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38dd12fa5c1e4f9f98cd4d0928b45b30, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.065138 19145 raft_consensus.cc:2804] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:39.065327 19145 raft_consensus.cc:697] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Becoming Leader. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Running, Role: LEADER
I20260430 20:02:39.065656 19145 consensus_queue.cc:237] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } }
I20260430 20:02:39.069341 18622 catalog_manager.cc:5671] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "8172e68f2b984b06ae18a352a74ed834" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:39.082736 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.078670 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.10.46.2:41743)
0430 20:02:39.078858 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.078861 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.078875 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:39.078957 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.078960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.079012 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.079105 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:39.079558 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.080143 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.081111 (+ 968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.081300 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.081354 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.082085 (+ 731us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.082307 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.082556 (+ 249us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.082611 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260430 20:02:39.088819 19065 raft_consensus.cc:1275] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.089257 18781 raft_consensus.cc:1275] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.089596 19145 consensus_queue.cc:1048] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.089906 19147 consensus_queue.cc:1048] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.097190 19152 mvcc.cc:204] Tried to move back new op lower bound from 7280965054819573760 to 7280965054735994880. Current Snapshot: MvccSnapshot[applied={T|T < 7280965054819573760}]
I20260430 20:02:39.099160 19154 mvcc.cc:204] Tried to move back new op lower bound from 7280965054819573760 to 7280965054735994880. Current Snapshot: MvccSnapshot[applied={T|T < 7280965054819573760}]
I20260430 20:02:39.100049 19153 mvcc.cc:204] Tried to move back new op lower bound from 7280965054819573760 to 7280965054735994880. Current Snapshot: MvccSnapshot[applied={T|T < 7280965054819573760}]
I20260430 20:02:39.108053 18620 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:39.108263 18620 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:39.111018 18620 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Sending DeleteTablet for 3 replicas of tablet 9ab438d0a11241908a7ee0b1bd85e7b2
I20260430 20:02:39.112066 18761 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33468
I20260430 20:02:39.112217 19045 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:42416
I20260430 20:02:39.112738 18903 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ab438d0a11241908a7ee0b1bd85e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260430 20:02:39.113024 19163 tablet_replica.cc:333] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: stopping tablet replica
I20260430 20:02:39.113025 18620 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 20:02:39.113190 19164 tablet_replica.cc:333] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: stopping tablet replica
I20260430 20:02:39.113348 19163 raft_consensus.cc:2243] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:39.113423 19164 raft_consensus.cc:2243] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:39.113703 19163 raft_consensus.cc:2272] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:39.113816 19164 raft_consensus.cc:2272] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260430 20:02:39.113960 18620 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:39.114094 19162 tablet_replica.cc:333] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: stopping tablet replica
I20260430 20:02:39.114354 19162 raft_consensus.cc:2243] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:39.114668 19162 raft_consensus.cc:2272] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:39.115401 19164 ts_tablet_manager.cc:1916] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.115994 19162 ts_tablet_manager.cc:1916] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.118100 19164 ts_tablet_manager.cc:1929] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:39.118196 19164 log.cc:1199] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/9ab438d0a11241908a7ee0b1bd85e7b2
I20260430 20:02:39.118496 19164 ts_tablet_manager.cc:1950] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 8172e68f2b984b06ae18a352a74ed834: Deleting consensus metadata
I20260430 20:02:39.118465 19162 ts_tablet_manager.cc:1929] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:39.118567 19162 log.cc:1199] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/9ab438d0a11241908a7ee0b1bd85e7b2
I20260430 20:02:39.118858 19162 ts_tablet_manager.cc:1950] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting consensus metadata
I20260430 20:02:39.120615 18606 catalog_manager.cc:5002] TS 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743): tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (table test-table [id=19a6937e0fd24d3c92128e02cb08b80b]) successfully deleted
I20260430 20:02:39.120784 18606 catalog_manager.cc:5002] TS 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923): tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (table test-table [id=19a6937e0fd24d3c92128e02cb08b80b]) successfully deleted
I20260430 20:02:39.120972 19163 ts_tablet_manager.cc:1916] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.121171 19045 tablet_service.cc:1511] Processing CreateTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 (DEFAULT_TABLE table=test-table [id=b4acec77ef324f95b125f50355024024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.121235 18903 tablet_service.cc:1511] Processing CreateTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 (DEFAULT_TABLE table=test-table [id=b4acec77ef324f95b125f50355024024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.121471 19045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9329bc203041a1a44d9e5e49fadce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.121529 18903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9329bc203041a1a44d9e5e49fadce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.123497 19163 ts_tablet_manager.cc:1929] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:39.123594 19163 log.cc:1199] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/9ab438d0a11241908a7ee0b1bd85e7b2
I20260430 20:02:39.123865 19141 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Bootstrap starting.
I20260430 20:02:39.123909 19140 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Bootstrap starting.
I20260430 20:02:39.123929 19163 ts_tablet_manager.cc:1950] T 9ab438d0a11241908a7ee0b1bd85e7b2 P 1c32797c40f74a11be42c0824898e583: Deleting consensus metadata
I20260430 20:02:39.124528 18761 tablet_service.cc:1511] Processing CreateTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 (DEFAULT_TABLE table=test-table [id=b4acec77ef324f95b125f50355024024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.124847 18761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9329bc203041a1a44d9e5e49fadce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.125195 19140 tablet_bootstrap.cc:654] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.126168 19141 tablet_bootstrap.cc:654] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.127342 19140 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: No bootstrap required, opened a new log
I20260430 20:02:39.127457 19140 ts_tablet_manager.cc:1403] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:39.127810 19139 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Bootstrap starting.
I20260430 20:02:39.128988 19139 tablet_bootstrap.cc:654] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.129055 19140 raft_consensus.cc:359] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.129191 19140 raft_consensus.cc:385] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.129232 19140 raft_consensus.cc:740] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.129380 19140 consensus_queue.cc:260] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.129686 19140 ts_tablet_manager.cc:1434] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 20:02:39.129741 18606 catalog_manager.cc:5002] TS 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787): tablet 9ab438d0a11241908a7ee0b1bd85e7b2 (table test-table [id=19a6937e0fd24d3c92128e02cb08b80b]) successfully deleted
I20260430 20:02:39.130896 19141 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: No bootstrap required, opened a new log
I20260430 20:02:39.131026 19141 ts_tablet_manager.cc:1403] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260430 20:02:39.131630 19141 raft_consensus.cc:359] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.131759 19141 raft_consensus.cc:385] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.131827 19141 raft_consensus.cc:740] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.131961 19141 consensus_queue.cc:260] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.132081 19139 tablet_bootstrap.cc:492] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: No bootstrap required, opened a new log
I20260430 20:02:39.132169 19139 ts_tablet_manager.cc:1403] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 20:02:39.132939 19141 ts_tablet_manager.cc:1434] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260430 20:02:39.132948 19139 raft_consensus.cc:359] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.133144 19139 raft_consensus.cc:385] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.133190 19139 raft_consensus.cc:740] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.133395 19139 consensus_queue.cc:260] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.133904 19139 ts_tablet_manager.cc:1434] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260430 20:02:39.172689 18970 tablet.cc:2404] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 20:02:39.204391 19112 tablet.cc:2404] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:02:39.279274 19147 raft_consensus.cc:493] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:39.279452 19147 raft_consensus.cc:515] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.279995 19147 leader_election.cc:290] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:39.280604 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a9329bc203041a1a44d9e5e49fadce4" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" is_pre_election: true
I20260430 20:02:39.280629 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a9329bc203041a1a44d9e5e49fadce4" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583" is_pre_election: true
I20260430 20:02:39.280782 18781 raft_consensus.cc:2468] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.280782 19065 raft_consensus.cc:2468] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.281096 18854 leader_election.cc:304] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32797c40f74a11be42c0824898e583, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.281318 19147 raft_consensus.cc:2804] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:39.281383 19147 raft_consensus.cc:493] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:39.281440 19147 raft_consensus.cc:3060] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.282461 19147 raft_consensus.cc:515] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.282887 19147 leader_election.cc:290] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Requested vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:39.283301 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a9329bc203041a1a44d9e5e49fadce4" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583"
I20260430 20:02:39.283293 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a9329bc203041a1a44d9e5e49fadce4" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
I20260430 20:02:39.283423 18781 raft_consensus.cc:3060] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.283423 19065 raft_consensus.cc:3060] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.284564 18781 raft_consensus.cc:2468] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:39.284564 19065 raft_consensus.cc:2468] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:39.284905 18854 leader_election.cc:304] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38dd12fa5c1e4f9f98cd4d0928b45b30, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.285084 19147 raft_consensus.cc:2804] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:39.285171 19147 raft_consensus.cc:697] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Becoming Leader. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Running, Role: LEADER
I20260430 20:02:39.285344 19147 consensus_queue.cc:237] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.287171 18621 catalog_manager.cc:5671] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "8172e68f2b984b06ae18a352a74ed834" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } health_report { overall_health: UNKNOWN } } }
Apr 30 20:02:39 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579357, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.62@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:39.326905 19174 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.318115 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:35705 (local address 127.0.0.1:53612)
0430 20:02:39.318485 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.318497 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.318666 (+ 169us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.318914 (+ 248us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.318917 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.319381 (+ 464us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:39.319657 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.319666 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.320531 (+ 865us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.320535 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.320898 (+ 363us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.320904 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.321036 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.321681 (+ 645us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:39.321695 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:39.323221 (+ 1526us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:39.325375 (+ 2154us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:39.325383 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:39.325387 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:39.325605 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:39.325867 (+ 262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:39.325871 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:39.325872 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:39.325930 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:39.326310 (+ 380us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:39.326318 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:39.326416 (+ 98us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.326531 (+ 115us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.326696 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":167,"threads_started":1}
I20260430 20:02:39.326905 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.318232 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.10.46.62:35705)
0430 20:02:39.318407 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.318412 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.318603 (+ 191us) server_negotiation.cc:408] Connection header received
0430 20:02:39.318764 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.318767 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.318811 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.318898 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:39.319835 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.320410 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.321009 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.321185 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.323359 (+ 2174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:39.323388 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:39.323390 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:39.323412 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:39.325132 (+ 1720us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.325717 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.325722 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.325724 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.325771 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.326033 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.326036 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.326038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.326193 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:39.326300 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.326605 (+ 305us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.326753 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":78}
I20260430 20:02:39.331390 18621 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:53612:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 20:02:39.332170 18621 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:39.338438 18761 tablet_service.cc:1511] Processing CreateTablet for tablet 2338523f177d4afbb46a62f0500a679e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0db2e3ec33574f9caf5f4e7e2b998869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:02:39.338470 18903 tablet_service.cc:1511] Processing CreateTablet for tablet 2338523f177d4afbb46a62f0500a679e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0db2e3ec33574f9caf5f4e7e2b998869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:02:39.338617 19045 tablet_service.cc:1511] Processing CreateTablet for tablet 2338523f177d4afbb46a62f0500a679e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0db2e3ec33574f9caf5f4e7e2b998869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 20:02:39.338878 18761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338523f177d4afbb46a62f0500a679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.338874 18903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338523f177d4afbb46a62f0500a679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.338930 19045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338523f177d4afbb46a62f0500a679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.341346 19141 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: Bootstrap starting.
I20260430 20:02:39.341591 19140 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: Bootstrap starting.
I20260430 20:02:39.342401 19141 tablet_bootstrap.cc:654] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.342582 19140 tablet_bootstrap.cc:654] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.343412 19139 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: Bootstrap starting.
I20260430 20:02:39.343694 19141 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: No bootstrap required, opened a new log
I20260430 20:02:39.343818 19141 ts_tablet_manager.cc:1403] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260430 20:02:39.344326 19141 raft_consensus.cc:359] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.344470 19141 raft_consensus.cc:385] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.344504 19141 raft_consensus.cc:740] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.344647 19141 consensus_queue.cc:260] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.344821 19139 tablet_bootstrap.cc:654] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.344856 19140 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: No bootstrap required, opened a new log
I20260430 20:02:39.344960 19140 ts_tablet_manager.cc:1403] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:39.344959 19141 ts_tablet_manager.cc:1434] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:39.345259 19147 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.345391 19147 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.345497 19140 raft_consensus.cc:359] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.345608 19140 raft_consensus.cc:385] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.345633 19140 raft_consensus.cc:740] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.345777 19140 consensus_queue.cc:260] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.346057 19140 ts_tablet_manager.cc:1434] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:02:39.346031 19146 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.346135 19146 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.346737 19139 tablet_bootstrap.cc:492] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: No bootstrap required, opened a new log
I20260430 20:02:39.346859 19139 ts_tablet_manager.cc:1403] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 20:02:39.347430 19139 raft_consensus.cc:359] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.347561 19139 raft_consensus.cc:385] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.347608 19139 raft_consensus.cc:740] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.347838 19139 consensus_queue.cc:260] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.348155 19139 ts_tablet_manager.cc:1434] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:39.348114 19148 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.348235 19148 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: This TxnStatusTablet replica's current role is: FOLLOWER
W20260430 20:02:39.378542 18828 tablet.cc:2404] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 20:02:39.453251 19148 raft_consensus.cc:493] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:39.453449 19148 raft_consensus.cc:515] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.454593 19148 leader_election.cc:290] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:39.458860 19178 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.454973 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:41743 (local address 127.10.46.1:59601)
0430 20:02:39.455397 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.455416 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.455633 (+ 217us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.456014 (+ 381us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.456017 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.456043 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.456370 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.456377 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.457667 (+ 1290us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.457672 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.458490 (+ 818us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.458498 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.458628 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.458644 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.458682 (+ 38us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.458728 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260430 20:02:39.459482 19179 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.455522 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:46787 (local address 127.10.46.1:49487)
0430 20:02:39.455871 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.455885 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.455983 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.456288 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.456291 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.456319 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:39.456905 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.456917 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.458222 (+ 1305us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.458226 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.459077 (+ 851us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.459087 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.459220 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.459235 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.459287 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.459340 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":82,"threads_started":1}
I20260430 20:02:39.459554 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.455062 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:59601 (local address 127.10.46.2:41743)
0430 20:02:39.455239 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.455244 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.455529 (+ 285us) server_negotiation.cc:408] Connection header received
0430 20:02:39.455830 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.455834 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.455897 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.455981 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.456611 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.457476 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.458692 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.459261 (+ 569us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.459316 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.459369 (+ 53us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.459433 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260430 20:02:39.459980 19134 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.455654 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:49487 (local address 127.10.46.3:46787)
0430 20:02:39.455853 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.455856 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.455981 (+ 125us) server_negotiation.cc:408] Connection header received
0430 20:02:39.456066 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.456068 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.456118 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.456207 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:39.457172 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.458046 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.459214 (+ 1168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.459649 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.459692 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.459751 (+ 59us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.459838 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 20:02:39.460139 18922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2338523f177d4afbb46a62f0500a679e" candidate_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8172e68f2b984b06ae18a352a74ed834" is_pre_election: true
I20260430 20:02:39.460399 18922 raft_consensus.cc:2468] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38dd12fa5c1e4f9f98cd4d0928b45b30 in term 0.
I20260430 20:02:39.460407 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2338523f177d4afbb46a62f0500a679e" candidate_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583" is_pre_election: true
I20260430 20:02:39.460549 19065 raft_consensus.cc:2468] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38dd12fa5c1e4f9f98cd4d0928b45b30 in term 0.
I20260430 20:02:39.460882 18712 leader_election.cc:304] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38dd12fa5c1e4f9f98cd4d0928b45b30, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.461128 19148 raft_consensus.cc:2804] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:39.461194 19148 raft_consensus.cc:493] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:39.461232 19148 raft_consensus.cc:3060] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.462173 19148 raft_consensus.cc:515] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.462598 19148 leader_election.cc:290] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [CANDIDATE]: Term 1 election: Requested vote from peers 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:39.462991 18922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2338523f177d4afbb46a62f0500a679e" candidate_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8172e68f2b984b06ae18a352a74ed834"
I20260430 20:02:39.463052 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2338523f177d4afbb46a62f0500a679e" candidate_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583"
I20260430 20:02:39.463138 18922 raft_consensus.cc:3060] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.463181 19065 raft_consensus.cc:3060] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.464187 18922 raft_consensus.cc:2468] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38dd12fa5c1e4f9f98cd4d0928b45b30 in term 1.
I20260430 20:02:39.464205 19065 raft_consensus.cc:2468] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38dd12fa5c1e4f9f98cd4d0928b45b30 in term 1.
I20260430 20:02:39.464534 18712 leader_election.cc:304] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38dd12fa5c1e4f9f98cd4d0928b45b30, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.464758 19148 raft_consensus.cc:2804] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:39.464993 19148 raft_consensus.cc:697] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 LEADER]: Becoming Leader. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Running, Role: LEADER
I20260430 20:02:39.465286 19148 consensus_queue.cc:237] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.466513 19180 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: TxnStatusTablet state changed. Reason: New leader 38dd12fa5c1e4f9f98cd4d0928b45b30. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.466630 19180 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:02:39.466922 19182 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 20:02:39.467072 19182 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 20:02:39.468398 18621 catalog_manager.cc:5671] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1). New cstate: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:39.505380 19138 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.501624 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.10.46.1:38923)
0430 20:02:39.501943 (+ 319us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.501950 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.501975 (+ 25us) server_negotiation.cc:408] Connection header received
0430 20:02:39.502174 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.502180 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.502292 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.502383 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:39.502869 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.503366 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.504348 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.504482 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.504526 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.504929 (+ 403us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.505051 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.505160 (+ 109us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.505223 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260430 20:02:39.505390 19174 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.500902 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.0.0.1:33480)
0430 20:02:39.501486 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.501535 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.501928 (+ 393us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.502532 (+ 604us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.502534 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.502542 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:02:39.502738 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.502743 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.503488 (+ 745us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.503494 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.504052 (+ 558us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.504060 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.504199 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.504241 (+ 42us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.505036 (+ 795us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.505049 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.505149 (+ 100us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.505232 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260430 20:02:39.508972 19065 raft_consensus.cc:1275] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Refusing update from remote peer 38dd12fa5c1e4f9f98cd4d0928b45b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.509020 18922 raft_consensus.cc:1275] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Refusing update from remote peer 38dd12fa5c1e4f9f98cd4d0928b45b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.509618 19148 consensus_queue.cc:1048] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.509831 19180 consensus_queue.cc:1048] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.511796 19146 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: TxnStatusTablet state changed. Reason: New leader 38dd12fa5c1e4f9f98cd4d0928b45b30. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.511931 19146 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.512127 19147 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: TxnStatusTablet state changed. Reason: New leader 38dd12fa5c1e4f9f98cd4d0928b45b30. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.512233 19147 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.514984 19146 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.515105 19146 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 1c32797c40f74a11be42c0824898e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.515271 19147 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.515379 19147 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 8172e68f2b984b06ae18a352a74ed834: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 20:02:39.515420 19180 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.515539 19180 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:02:39.515885 19187 tablet_replica.cc:442] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } } }
I20260430 20:02:39.516562 19187 tablet_replica.cc:445] T 2338523f177d4afbb46a62f0500a679e P 38dd12fa5c1e4f9f98cd4d0928b45b30: This TxnStatusTablet replica's current role is: LEADER
I20260430 20:02:39.530998 19191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.528011 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.10.46.2:42483)
0430 20:02:39.528322 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.528338 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.528487 (+ 149us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.528764 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.528768 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.528777 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:02:39.529008 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.529013 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.529730 (+ 717us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.529736 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.530148 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.530154 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.530287 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.530332 (+ 45us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.530676 (+ 344us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.530685 (+ 9us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.530768 (+ 83us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.530837 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20260430 20:02:39.530987 19138 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.528128 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:42483 (local address 127.10.46.1:38923)
0430 20:02:39.528303 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.528308 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.528419 (+ 111us) server_negotiation.cc:408] Connection header received
0430 20:02:39.528603 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.528607 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.528668 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.528753 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:39.529118 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.529617 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.530265 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.530470 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.530509 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.530582 (+ 73us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.530669 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.530794 (+ 125us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.530836 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:02:39.539307 19065 raft_consensus.cc:1275] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.539367 18781 raft_consensus.cc:1275] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:39.539815 19147 consensus_queue.cc:1048] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.539990 19158 consensus_queue.cc:1048] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:39.567947 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.562201 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:47699 (local address 127.10.46.2:41743)
0430 20:02:39.562339 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.562344 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.564189 (+ 1845us) server_negotiation.cc:408] Connection header received
0430 20:02:39.564364 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.564368 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.564564 (+ 196us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.564678 (+ 114us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:39.565557 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.566238 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.567194 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.567373 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.567415 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.567496 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.567573 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.567726 (+ 153us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.567829 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260430 20:02:39.568503 19195 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.561846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:41743 (local address 127.10.46.1:47699)
0430 20:02:39.564065 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:39.564083 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:39.564206 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:39.565049 (+ 843us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:39.565053 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:39.565062 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:02:39.565418 (+ 356us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.565425 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.566351 (+ 926us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.566355 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:39.567048 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:39.567057 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.567181 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.567229 (+ 48us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:02:39.567604 (+ 375us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:02:39.567610 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 20:02:39.567683 (+ 73us) client_negotiation.cc:241] Negotiation successful
0430 20:02:39.567750 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"thread_start_us":111,"threads_started":1}
I20260430 20:02:39.601001 18620 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:39.601189 18620 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:39.604612 18620 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Sending DeleteTablet for 3 replicas of tablet 7a9329bc203041a1a44d9e5e49fadce4
I20260430 20:02:39.605712 18903 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260430 20:02:39.605832 19045 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:42416
I20260430 20:02:39.605945 19164 tablet_replica.cc:333] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: stopping tablet replica
I20260430 20:02:39.606069 19164 raft_consensus.cc:2243] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:39.606159 19163 tablet_replica.cc:333] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: stopping tablet replica
I20260430 20:02:39.606302 19164 raft_consensus.cc:2272] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:39.606308 19163 raft_consensus.cc:2243] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:39.606423 19163 raft_consensus.cc:2272] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:39.607642 18761 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a9329bc203041a1a44d9e5e49fadce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:39 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33468
I20260430 20:02:39.608042 19162 tablet_replica.cc:333] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: stopping tablet replica
I20260430 20:02:39.608057 19164 ts_tablet_manager.cc:1916] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.608176 19162 raft_consensus.cc:2243] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:39.608295 19162 raft_consensus.cc:2272] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:39.608709 19163 ts_tablet_manager.cc:1916] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.610822 19164 ts_tablet_manager.cc:1929] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:39.610893 19164 log.cc:1199] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/7a9329bc203041a1a44d9e5e49fadce4
I20260430 20:02:39.611142 19164 ts_tablet_manager.cc:1950] T 7a9329bc203041a1a44d9e5e49fadce4 P 8172e68f2b984b06ae18a352a74ed834: Deleting consensus metadata
I20260430 20:02:39.611158 19163 ts_tablet_manager.cc:1929] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:39.611213 19163 log.cc:1199] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/7a9329bc203041a1a44d9e5e49fadce4
I20260430 20:02:39.611402 19163 ts_tablet_manager.cc:1950] T 7a9329bc203041a1a44d9e5e49fadce4 P 1c32797c40f74a11be42c0824898e583: Deleting consensus metadata
I20260430 20:02:39.612308 19162 ts_tablet_manager.cc:1916] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:39.612437 18606 catalog_manager.cc:5002] TS 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787): tablet 7a9329bc203041a1a44d9e5e49fadce4 (table test-table [id=b4acec77ef324f95b125f50355024024]) successfully deleted
I20260430 20:02:39.612877 18606 catalog_manager.cc:5002] TS 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743): tablet 7a9329bc203041a1a44d9e5e49fadce4 (table test-table [id=b4acec77ef324f95b125f50355024024]) successfully deleted
I20260430 20:02:39.615427 19162 ts_tablet_manager.cc:1929] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:39.615554 19162 log.cc:1199] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/7a9329bc203041a1a44d9e5e49fadce4
I20260430 20:02:39.615864 19162 ts_tablet_manager.cc:1950] T 7a9329bc203041a1a44d9e5e49fadce4 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting consensus metadata
I20260430 20:02:39.616665 18606 catalog_manager.cc:5002] TS 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923): tablet 7a9329bc203041a1a44d9e5e49fadce4 (table test-table [id=b4acec77ef324f95b125f50355024024]) successfully deleted
Apr 30 20:02:39 dist-test-slave-0bdw krb5kdc[18574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 20:02:39 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:39 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
W20260430 20:02:39.641505 18671 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:39.633008 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.10.46.62:35705)
0430 20:02:39.633155 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.633177 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.633709 (+ 532us) server_negotiation.cc:408] Connection header received
0430 20:02:39.633935 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.633939 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.634000 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.634098 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:39.635531 (+ 1433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.636613 (+ 1082us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.637686 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.637924 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.641339 (+ 3415us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53626: BlockingRecv error: recv got EOF from 127.0.0.1:53626 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Apr 30 20:02:39 dist-test-slave-0bdw krb5kdc[18574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM
I20260430 20:02:39.654815 18671 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:39.644112 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.10.46.62:35705)
0430 20:02:39.644243 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:39.644249 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:39.644359 (+ 110us) server_negotiation.cc:408] Connection header received
0430 20:02:39.644592 (+ 233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:39.644596 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:39.644659 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:39.644772 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:39.645920 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.646720 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:39.647632 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:39.647880 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:39.650922 (+ 3042us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:39.650946 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:39.650949 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:39.650982 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:39.652695 (+ 1713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.653259 (+ 564us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.653262 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.653263 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.653337 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:39.653653 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:39.653656 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:39.653657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:39.653821 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:39.653944 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:39.654544 (+ 600us) server_negotiation.cc:300] Negotiation successful
0430 20:02:39.654674 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260430 20:02:39.658897 18620 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:02:39.659483 18620 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:39.668895 18903 tablet_service.cc:1511] Processing CreateTablet for tablet 526d2658703a47ae8fffe0dc8d774644 (DEFAULT_TABLE table=test-table [id=a0b051319c2d4f0baf5f37a3d1356923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.669281 18903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526d2658703a47ae8fffe0dc8d774644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.671989 19141 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Bootstrap starting.
I20260430 20:02:39.672163 18761 tablet_service.cc:1511] Processing CreateTablet for tablet 526d2658703a47ae8fffe0dc8d774644 (DEFAULT_TABLE table=test-table [id=a0b051319c2d4f0baf5f37a3d1356923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.672482 18761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526d2658703a47ae8fffe0dc8d774644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.673178 19141 tablet_bootstrap.cc:654] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.674995 19139 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Bootstrap starting.
I20260430 20:02:39.676052 19139 tablet_bootstrap.cc:654] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.677150 19045 tablet_service.cc:1511] Processing CreateTablet for tablet 526d2658703a47ae8fffe0dc8d774644 (DEFAULT_TABLE table=test-table [id=a0b051319c2d4f0baf5f37a3d1356923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:39.677484 19045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526d2658703a47ae8fffe0dc8d774644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:39.679764 19140 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Bootstrap starting.
I20260430 20:02:39.680779 19139 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: No bootstrap required, opened a new log
I20260430 20:02:39.680855 19140 tablet_bootstrap.cc:654] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:39.680872 19139 ts_tablet_manager.cc:1403] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260430 20:02:39.681340 19139 raft_consensus.cc:359] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.681458 19139 raft_consensus.cc:385] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.681488 19139 raft_consensus.cc:740] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.682164 19141 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: No bootstrap required, opened a new log
I20260430 20:02:39.682271 19140 tablet_bootstrap.cc:492] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: No bootstrap required, opened a new log
I20260430 20:02:39.682307 19141 ts_tablet_manager.cc:1403] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260430 20:02:39.682354 19140 ts_tablet_manager.cc:1403] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:39.682824 19140 raft_consensus.cc:359] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.682835 19141 raft_consensus.cc:359] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.682920 19140 raft_consensus.cc:385] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.682938 19141 raft_consensus.cc:385] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:39.682947 19140 raft_consensus.cc:740] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.682969 19141 raft_consensus.cc:740] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Initialized, Role: FOLLOWER
I20260430 20:02:39.683079 19140 consensus_queue.cc:260] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.683336 19140 ts_tablet_manager.cc:1434] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:39.684062 19139 consensus_queue.cc:260] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.684333 19141 consensus_queue.cc:260] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.686370 19139 ts_tablet_manager.cc:1434] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260430 20:02:39.687368 19141 ts_tablet_manager.cc:1434] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260430 20:02:39.996104 19145 raft_consensus.cc:493] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:39.996282 19145 raft_consensus.cc:515] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.996706 19145 leader_election.cc:290] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:39.997383 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "526d2658703a47ae8fffe0dc8d774644" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" is_pre_election: true
I20260430 20:02:39.997529 18781 raft_consensus.cc:2468] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.997641 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "526d2658703a47ae8fffe0dc8d774644" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583" is_pre_election: true
I20260430 20:02:39.997843 19065 raft_consensus.cc:2468] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 0.
I20260430 20:02:39.997972 18854 leader_election.cc:304] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38dd12fa5c1e4f9f98cd4d0928b45b30, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:39.998155 19145 raft_consensus.cc:2804] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:39.998240 19145 raft_consensus.cc:493] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:39.998289 19145 raft_consensus.cc:3060] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:39.999262 19145 raft_consensus.cc:515] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:39.999668 19145 leader_election.cc:290] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Requested vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787)
I20260430 20:02:40.000473 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "526d2658703a47ae8fffe0dc8d774644" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
I20260430 20:02:40.000522 19065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "526d2658703a47ae8fffe0dc8d774644" candidate_uuid: "8172e68f2b984b06ae18a352a74ed834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32797c40f74a11be42c0824898e583"
I20260430 20:02:40.000638 18781 raft_consensus.cc:3060] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:40.000772 19065 raft_consensus.cc:3060] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:40.001689 19065 raft_consensus.cc:2468] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:40.001700 18781 raft_consensus.cc:2468] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8172e68f2b984b06ae18a352a74ed834 in term 1.
I20260430 20:02:40.002205 18854 leader_election.cc:304] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32797c40f74a11be42c0824898e583, 8172e68f2b984b06ae18a352a74ed834; no voters:
I20260430 20:02:40.002511 19145 raft_consensus.cc:2804] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:40.002664 19145 raft_consensus.cc:697] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Becoming Leader. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Running, Role: LEADER
I20260430 20:02:40.002879 19145 consensus_queue.cc:237] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.005043 18622 catalog_manager.cc:5671] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2). New cstate: current_term: 1 leader_uuid: "8172e68f2b984b06ae18a352a74ed834" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20260430 20:02:40.065550 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.061519 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.10.46.2:41743)
0430 20:02:40.061743 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.061750 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.061778 (+ 28us) server_negotiation.cc:408] Connection header received
0430 20:02:40.061858 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.061862 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.062158 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.062461 (+ 303us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:40.063450 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.064006 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.064773 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.064972 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.065013 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.065094 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.065188 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.065335 (+ 147us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.065386 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 20:02:40.069644 19065 raft_consensus.cc:1275] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:40.069644 18781 raft_consensus.cc:1275] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Refusing update from remote peer 8172e68f2b984b06ae18a352a74ed834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:40.070423 19145 consensus_queue.cc:1048] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:40.070675 19147 consensus_queue.cc:1048] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:40.080188 18621 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:40.080334 18621 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:40.082631 18621 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Sending DeleteTablet for 3 replicas of tablet 526d2658703a47ae8fffe0dc8d774644
I20260430 20:02:40.083597 18903 tablet_service.cc:1558] Processing DeleteTablet for tablet 526d2658703a47ae8fffe0dc8d774644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260430 20:02:40.083636 18761 tablet_service.cc:1558] Processing DeleteTablet for tablet 526d2658703a47ae8fffe0dc8d774644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33468
I20260430 20:02:40.083858 19162 tablet_replica.cc:333] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: stopping tablet replica
I20260430 20:02:40.083863 19045 tablet_service.cc:1558] Processing DeleteTablet for tablet 526d2658703a47ae8fffe0dc8d774644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:42416
I20260430 20:02:40.083976 19162 raft_consensus.cc:2243] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:40.084040 19163 tablet_replica.cc:333] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: stopping tablet replica
I20260430 20:02:40.084052 19164 tablet_replica.cc:333] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: stopping tablet replica
I20260430 20:02:40.084115 19162 raft_consensus.cc:2272] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.084146 19163 raft_consensus.cc:2243] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:40.084146 19164 raft_consensus.cc:2243] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:40.084270 19163 raft_consensus.cc:2272] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.084355 19164 raft_consensus.cc:2272] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.084849 19162 ts_tablet_manager.cc:1916] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:40.084820 18621 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 20:02:40.085014 19163 ts_tablet_manager.cc:1916] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260430 20:02:40.085315 18621 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:02:40.086905 19162 ts_tablet_manager.cc:1929] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:40.086980 19162 log.cc:1199] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/526d2658703a47ae8fffe0dc8d774644
I20260430 20:02:40.087085 19164 ts_tablet_manager.cc:1916] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:40.087246 19162 ts_tablet_manager.cc:1950] T 526d2658703a47ae8fffe0dc8d774644 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting consensus metadata
I20260430 20:02:40.087297 19163 ts_tablet_manager.cc:1929] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:40.087368 19163 log.cc:1199] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/526d2658703a47ae8fffe0dc8d774644
I20260430 20:02:40.087625 19163 ts_tablet_manager.cc:1950] T 526d2658703a47ae8fffe0dc8d774644 P 1c32797c40f74a11be42c0824898e583: Deleting consensus metadata
I20260430 20:02:40.088011 18606 catalog_manager.cc:5002] TS 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923): tablet 526d2658703a47ae8fffe0dc8d774644 (table test-table [id=a0b051319c2d4f0baf5f37a3d1356923]) successfully deleted
I20260430 20:02:40.089274 19164 ts_tablet_manager.cc:1929] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:02:40.089341 19164 log.cc:1199] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/526d2658703a47ae8fffe0dc8d774644
I20260430 20:02:40.089591 19164 ts_tablet_manager.cc:1950] T 526d2658703a47ae8fffe0dc8d774644 P 8172e68f2b984b06ae18a352a74ed834: Deleting consensus metadata
I20260430 20:02:40.090411 18606 catalog_manager.cc:5002] TS 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787): tablet 526d2658703a47ae8fffe0dc8d774644 (table test-table [id=a0b051319c2d4f0baf5f37a3d1356923]) successfully deleted
I20260430 20:02:40.090694 18606 catalog_manager.cc:5002] TS 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743): tablet 526d2658703a47ae8fffe0dc8d774644 (table test-table [id=a0b051319c2d4f0baf5f37a3d1356923]) successfully deleted
I20260430 20:02:40.094041 18761 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 (DEFAULT_TABLE table=test-table [id=67e10d4babf34a17be333da2cd5b844e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:40.094133 18903 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 (DEFAULT_TABLE table=test-table [id=67e10d4babf34a17be333da2cd5b844e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:40.094399 18903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf011b13ff4a85b3a9f406b6cad338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:40.094412 18761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf011b13ff4a85b3a9f406b6cad338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:40.094460 19045 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 (DEFAULT_TABLE table=test-table [id=67e10d4babf34a17be333da2cd5b844e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:02:40.094745 19045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf011b13ff4a85b3a9f406b6cad338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:40.096822 19141 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Bootstrap starting.
I20260430 20:02:40.096822 19139 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Bootstrap starting.
I20260430 20:02:40.097748 19141 tablet_bootstrap.cc:654] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:40.097748 19139 tablet_bootstrap.cc:654] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:40.098345 19140 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Bootstrap starting.
I20260430 20:02:40.099140 19139 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: No bootstrap required, opened a new log
I20260430 20:02:40.099237 19139 ts_tablet_manager.cc:1403] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260430 20:02:40.099269 19140 tablet_bootstrap.cc:654] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:40.099536 19141 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: No bootstrap required, opened a new log
I20260430 20:02:40.099622 19141 ts_tablet_manager.cc:1403] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260430 20:02:40.099740 19139 raft_consensus.cc:359] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.099915 19139 raft_consensus.cc:385] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:40.099960 19139 raft_consensus.cc:740] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38dd12fa5c1e4f9f98cd4d0928b45b30, State: Initialized, Role: FOLLOWER
I20260430 20:02:40.100087 19139 consensus_queue.cc:260] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.100113 19141 raft_consensus.cc:359] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.100245 19141 raft_consensus.cc:385] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:40.100296 19141 raft_consensus.cc:740] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8172e68f2b984b06ae18a352a74ed834, State: Initialized, Role: FOLLOWER
I20260430 20:02:40.100435 19141 consensus_queue.cc:260] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.100512 19139 ts_tablet_manager.cc:1434] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260430 20:02:40.100716 19141 ts_tablet_manager.cc:1434] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 20:02:40.100863 19140 tablet_bootstrap.cc:492] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: No bootstrap required, opened a new log
I20260430 20:02:40.100958 19140 ts_tablet_manager.cc:1403] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260430 20:02:40.101442 19140 raft_consensus.cc:359] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.101586 19140 raft_consensus.cc:385] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:40.101620 19140 raft_consensus.cc:740] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Initialized, Role: FOLLOWER
I20260430 20:02:40.101758 19140 consensus_queue.cc:260] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.102041 19140 ts_tablet_manager.cc:1434] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260430 20:02:40.390511 19146 raft_consensus.cc:493] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:02:40.390740 19146 raft_consensus.cc:515] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.392722 19146 leader_election.cc:290] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743)
I20260430 20:02:40.396694 19219 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.392832 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.10.46.3:38953)
0430 20:02:40.393136 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:40.393154 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:40.393302 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:40.394050 (+ 748us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:40.394056 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:40.394109 (+ 53us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:40.394363 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.394369 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.395193 (+ 824us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.395200 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:40.396208 (+ 1008us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.396219 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.396368 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.396388 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:02:40.396445 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:02:40.396510 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"spinlock_wait_cycles":5376,"thread_start_us":105,"threads_started":1}
I20260430 20:02:40.397145 19220 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.392928 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:38953 (local address 127.10.46.1:38923)
0430 20:02:40.393161 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.393166 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.393318 (+ 152us) server_negotiation.cc:408] Connection header received
0430 20:02:40.393728 (+ 410us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.393733 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.393799 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.393883 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:40.394493 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.395073 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.396370 (+ 1297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.396879 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.396915 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.396972 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.397025 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260430 20:02:40.397141 19137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.393357 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:42965 (local address 127.10.46.2:41743)
0430 20:02:40.393777 (+ 420us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.393783 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.393801 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:02:40.393859 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.393864 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.393915 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.393997 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:02:40.394519 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.395275 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.396431 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.396879 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.396930 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.396975 (+ 45us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.397025 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260430 20:02:40.396713 19221 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.393238 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:41743 (local address 127.10.46.3:42965)
0430 20:02:40.393486 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:40.393500 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:40.393598 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:40.394028 (+ 430us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:40.394031 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:40.394049 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:02:40.394381 (+ 332us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.394387 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.395415 (+ 1028us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.395418 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:40.396259 (+ 841us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.396267 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.396423 (+ 156us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.396440 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 20:02:40.396488 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 20:02:40.396533 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"spinlock_wait_cycles":5248,"thread_start_us":67,"threads_started":1}
I20260430 20:02:40.397811 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf011b13ff4a85b3a9f406b6cad338" candidate_uuid: "1c32797c40f74a11be42c0824898e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" is_pre_election: true
I20260430 20:02:40.397994 18781 raft_consensus.cc:2468] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c32797c40f74a11be42c0824898e583 in term 0.
I20260430 20:02:40.398068 18922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf011b13ff4a85b3a9f406b6cad338" candidate_uuid: "1c32797c40f74a11be42c0824898e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8172e68f2b984b06ae18a352a74ed834" is_pre_election: true
I20260430 20:02:40.398299 18922 raft_consensus.cc:2468] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c32797c40f74a11be42c0824898e583 in term 0.
I20260430 20:02:40.398684 18996 leader_election.cc:304] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32797c40f74a11be42c0824898e583, 38dd12fa5c1e4f9f98cd4d0928b45b30; no voters:
I20260430 20:02:40.399006 19146 raft_consensus.cc:2804] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:02:40.399076 19146 raft_consensus.cc:493] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:02:40.399114 19146 raft_consensus.cc:3060] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:40.400108 19146 raft_consensus.cc:515] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.400518 19146 leader_election.cc:290] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [CANDIDATE]: Term 1 election: Requested vote from peers 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923), 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743)
I20260430 20:02:40.401017 18781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf011b13ff4a85b3a9f406b6cad338" candidate_uuid: "1c32797c40f74a11be42c0824898e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30"
I20260430 20:02:40.401018 18922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf011b13ff4a85b3a9f406b6cad338" candidate_uuid: "1c32797c40f74a11be42c0824898e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8172e68f2b984b06ae18a352a74ed834"
I20260430 20:02:40.401154 18781 raft_consensus.cc:3060] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:40.401154 18922 raft_consensus.cc:3060] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:40.402241 18781 raft_consensus.cc:2468] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c32797c40f74a11be42c0824898e583 in term 1.
I20260430 20:02:40.402241 18922 raft_consensus.cc:2468] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c32797c40f74a11be42c0824898e583 in term 1.
I20260430 20:02:40.402678 18996 leader_election.cc:304] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32797c40f74a11be42c0824898e583, 38dd12fa5c1e4f9f98cd4d0928b45b30; no voters:
I20260430 20:02:40.402899 19146 raft_consensus.cc:2804] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:40.403129 19146 raft_consensus.cc:697] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 1 LEADER]: Becoming Leader. State: Replica: 1c32797c40f74a11be42c0824898e583, State: Running, Role: LEADER
I20260430 20:02:40.403505 19146 consensus_queue.cc:237] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } }
I20260430 20:02:40.406824 18621 catalog_manager.cc:5671] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c32797c40f74a11be42c0824898e583 (127.10.46.3). New cstate: current_term: 1 leader_uuid: "1c32797c40f74a11be42c0824898e583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c32797c40f74a11be42c0824898e583" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 46787 } health_report { overall_health: HEALTHY } } }
I20260430 20:02:40.478698 19226 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.474669 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.10.46.3:46787)
0430 20:02:40.474974 (+ 305us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.474978 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.474990 (+ 12us) server_negotiation.cc:408] Connection header received
0430 20:02:40.475068 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.475071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.475146 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.475254 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:40.475895 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.476527 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.477396 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.477579 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.477633 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.478037 (+ 404us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.478131 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.478419 (+ 288us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.478548 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":142,"threads_started":1}
I20260430 20:02:40.489966 19229 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.486209 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:38923 (local address 127.10.46.3:55789)
0430 20:02:40.486637 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:40.486651 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:40.486760 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:40.487228 (+ 468us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:40.487232 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:40.487241 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:02:40.487539 (+ 298us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.487544 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.488648 (+ 1104us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.488651 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:40.489117 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.489124 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.489231 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.489268 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.489682 (+ 414us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.489688 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 20:02:40.489760 (+ 72us) client_negotiation.cc:241] Negotiation successful
0430 20:02:40.489823 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":222,"threads_started":1}
I20260430 20:02:40.489965 19220 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.486366 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:55789 (local address 127.10.46.1:38923)
0430 20:02:40.486507 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.486510 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.486878 (+ 368us) server_negotiation.cc:408] Connection header received
0430 20:02:40.486992 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.486996 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.487075 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.487161 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:40.487813 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.488503 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.489281 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.489428 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.489469 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.489553 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.489624 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.489814 (+ 190us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.489860 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 20:02:40.497179 18922 raft_consensus.cc:1275] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Refusing update from remote peer 1c32797c40f74a11be42c0824898e583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:40.497223 18781 raft_consensus.cc:1275] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Refusing update from remote peer 1c32797c40f74a11be42c0824898e583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:02:40.497880 19222 consensus_queue.cc:1048] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38dd12fa5c1e4f9f98cd4d0928b45b30" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:40.498076 19146 consensus_queue.cc:1048] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8172e68f2b984b06ae18a352a74ed834" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:02:40.521370 19226 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.517043 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:36461 (local address 127.10.46.3:46787)
0430 20:02:40.517163 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:40.517168 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:40.517431 (+ 263us) server_negotiation.cc:408] Connection header received
0430 20:02:40.517881 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:40.517889 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:40.517990 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:40.518111 (+ 121us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:02:40.518642 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.519300 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.520166 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.520489 (+ 323us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.520546 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.520634 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.520877 (+ 243us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:40.521159 (+ 282us) server_negotiation.cc:300] Negotiation successful
0430 20:02:40.521220 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260430 20:02:40.521373 19234 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:40.516985 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:46787 (local address 127.10.46.1:36461)
0430 20:02:40.517236 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:40.517253 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:40.517579 (+ 326us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:40.518141 (+ 562us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:40.518145 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:40.518159 (+ 14us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 20:02:40.518506 (+ 347us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.518516 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.519423 (+ 907us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:40.519427 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:40.519960 (+ 533us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:40.519975 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:40.520144 (+ 169us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:40.520184 (+ 40us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 20:02:40.520912 (+ 728us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 20:02:40.520920 (+ 8us) client_negotiation.cc:770] Sending connection context
0430 20:02:40.521134 (+ 214us) client_negotiation.cc:241] Negotiation successful
0430 20:02:40.521234 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260430 20:02:40.546417 18621 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:40.546627 18621 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:02:40.549021 18621 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 02deb7bf8cf6468c81da741a98bf5297: Sending DeleteTablet for 3 replicas of tablet 9bbf011b13ff4a85b3a9f406b6cad338
I20260430 20:02:40.549751 18761 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33468
I20260430 20:02:40.549878 19045 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:42416
I20260430 20:02:40.549892 18903 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bbf011b13ff4a85b3a9f406b6cad338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:02:40 UTC) from {username='oryx', principal='oryx/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260430 20:02:40.549973 19162 tablet_replica.cc:333] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: stopping tablet replica
I20260430 20:02:40.550043 19163 tablet_replica.cc:333] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: stopping tablet replica
I20260430 20:02:40.550107 19164 tablet_replica.cc:333] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: stopping tablet replica
I20260430 20:02:40.550103 19162 raft_consensus.cc:2243] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:40.550132 19163 raft_consensus.cc:2243] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:02:40.550217 19164 raft_consensus.cc:2243] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:02:40.550225 19162 raft_consensus.cc:2272] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.550348 19164 raft_consensus.cc:2272] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.550438 19163 raft_consensus.cc:2272] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:02:40.551278 19163 ts_tablet_manager.cc:1916] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:40.551278 19162 ts_tablet_manager.cc:1916] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:40.551643 19164 ts_tablet_manager.cc:1916] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:02:40.553669 19162 ts_tablet_manager.cc:1929] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:40.553669 19163 ts_tablet_manager.cc:1929] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:40.553745 19162 log.cc:1199] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-0/wal/wals/9bbf011b13ff4a85b3a9f406b6cad338
I20260430 20:02:40.553743 19163 log.cc:1199] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/9bbf011b13ff4a85b3a9f406b6cad338
I20260430 20:02:40.554008 19163 ts_tablet_manager.cc:1950] T 9bbf011b13ff4a85b3a9f406b6cad338 P 1c32797c40f74a11be42c0824898e583: Deleting consensus metadata
I20260430 20:02:40.554008 19162 ts_tablet_manager.cc:1950] T 9bbf011b13ff4a85b3a9f406b6cad338 P 38dd12fa5c1e4f9f98cd4d0928b45b30: Deleting consensus metadata
I20260430 20:02:40.554044 19164 ts_tablet_manager.cc:1929] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 20:02:40.554121 19164 log.cc:1199] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/9bbf011b13ff4a85b3a9f406b6cad338
I20260430 20:02:40.554385 19164 ts_tablet_manager.cc:1950] T 9bbf011b13ff4a85b3a9f406b6cad338 P 8172e68f2b984b06ae18a352a74ed834: Deleting consensus metadata
I20260430 20:02:40.555341 18606 catalog_manager.cc:5002] TS 1c32797c40f74a11be42c0824898e583 (127.10.46.3:46787): tablet 9bbf011b13ff4a85b3a9f406b6cad338 (table test-table [id=67e10d4babf34a17be333da2cd5b844e]) successfully deleted
I20260430 20:02:40.555567 18606 catalog_manager.cc:5002] TS 38dd12fa5c1e4f9f98cd4d0928b45b30 (127.10.46.1:38923): tablet 9bbf011b13ff4a85b3a9f406b6cad338 (table test-table [id=67e10d4babf34a17be333da2cd5b844e]) successfully deleted
I20260430 20:02:40.556066 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18696
I20260430 20:02:40.561000 18606 catalog_manager.cc:5002] TS 8172e68f2b984b06ae18a352a74ed834 (127.10.46.2:41743): tablet 9bbf011b13ff4a85b3a9f406b6cad338 (table test-table [id=67e10d4babf34a17be333da2cd5b844e]) successfully deleted
I20260430 20:02:40.566102 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18838
I20260430 20:02:40.573870 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18980
I20260430 20:02:40.580931 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 18590
2026-04-30T20:02:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5059 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:40 dist-test-slave-0bdw krb5kdc[19239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:40 dist-test-slave-0bdw krb5kdc[19239](info): set up 2 sockets
Apr 30 20:02:40 dist-test-slave-0bdw krb5kdc[19239](info): commencing operation
krb5kdc: starting...
W20260430 20:02:42.637109 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.032s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:42 dist-test-slave-0bdw krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:42.788053 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35251
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:42.894266 19255 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:42.894503 19255 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:42.894562 19255 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:42.897993 19255 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:42.898068 19255 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:42.898092 19255 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:42.898111 19255 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:42.898129 19255 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:42.902665 19255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:42.903865 19255 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:42.904680 19255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:42.911067 19263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:42.911151 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:42.911151 19260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:42.911407 19255 server_base.cc:1061] running on GCE node
I20260430 20:02:42.912027 19255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:42.913040 19255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:42.914337 19255 hybrid_clock.cc:648] HybridClock initialized: now 1777579362914279 us; error 66 us; skew 500 ppm
W20260430 20:02:42.918169 19255 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-04-30T20:02:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2343 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:02:42 dist-test-slave-0bdw krb5kdc[19268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:02:42 dist-test-slave-0bdw krb5kdc[19268](info): set up 2 sockets
Apr 30 20:02:42 dist-test-slave-0bdw krb5kdc[19268](info): commencing operation
krb5kdc: starting...
W20260430 20:02:44.989638 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.041s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:02:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:45.144924 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41139
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:41139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:45.256847 19284 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:45.257113 19284 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:45.257171 19284 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:45.261680 19284 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:45.261777 19284 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:45.261802 19284 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:45.261822 19284 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:45.261840 19284 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:45.267014 19284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:41139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:45.268260 19284 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:45.269090 19284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:45.275453 19289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:45.275450 19292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:45.275461 19290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:45.275620 19284 server_base.cc:1061] running on GCE node
I20260430 20:02:45.276468 19284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:45.277436 19284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:45.278618 19284 hybrid_clock.cc:648] HybridClock initialized: now 1777579365278546 us; error 107 us; skew 500 ppm
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:45.282439 19284 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:02:45.283725 19284 webserver.cc:492] Webserver started at http://127.10.46.62:44671/ using document root <none> and password file <none>
I20260430 20:02:45.284379 19284 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:45.284462 19284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:45.284687 19284 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:45.286556 19284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "f08cae824112450cb2fac043215188e0"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "6906fd845933808855314ffc1bf82c19"
server_key_iv: "ee573ca34a493738bf036e91c5d66ccf"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.287098 19284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "f08cae824112450cb2fac043215188e0"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "6906fd845933808855314ffc1bf82c19"
server_key_iv: "ee573ca34a493738bf036e91c5d66ccf"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.291070 19284 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260430 20:02:45.293937 19299 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.295600 19284 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:02:45.295851 19284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "f08cae824112450cb2fac043215188e0"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "6906fd845933808855314ffc1bf82c19"
server_key_iv: "ee573ca34a493738bf036e91c5d66ccf"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.296010 19284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:45.310309 19284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:45.314013 19284 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:45.314267 19284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:45.323680 19284 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:41139
I20260430 20:02:45.323676 19351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:41139 every 8 connection(s)
I20260430 20:02:45.324880 19284 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:45.328003 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:02:45.332121 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19284
I20260430 20:02:45.332314 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:02:45.332597 10424 external_mini_cluster.cc:1468] Setting key 432cd7ae7319aaa27f1b65d631d20633
I20260430 20:02:45.334899 19352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Bootstrap starting.
I20260430 20:02:45.338274 19352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Neither blocks nor log segments found. Creating new log.
I20260430 20:02:45.339355 19352 log.cc:826] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:45.342801 19352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: No bootstrap required, opened a new log
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:45.345997 19352 raft_consensus.cc:359] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:45.346318 19352 raft_consensus.cc:385] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:02:45.346398 19352 raft_consensus.cc:740] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08cae824112450cb2fac043215188e0, State: Initialized, Role: FOLLOWER
I20260430 20:02:45.346920 19352 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:45.347090 19352 raft_consensus.cc:399] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:45.347150 19352 raft_consensus.cc:493] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:45.347262 19352 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:02:45.348453 19352 raft_consensus.cc:515] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:45.348863 19352 leader_election.cc:304] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08cae824112450cb2fac043215188e0; no voters:
I20260430 20:02:45.349328 19352 leader_election.cc:290] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:02:45.349766 19357 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:02:45.350032 19357 raft_consensus.cc:697] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 LEADER]: Becoming Leader. State: Replica: f08cae824112450cb2fac043215188e0, State: Running, Role: LEADER
I20260430 20:02:45.350431 19355 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:45.334468 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.10.46.62:41139)
0430 20:02:45.335278 (+ 810us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:45.335288 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:45.335326 (+ 38us) server_negotiation.cc:408] Connection header received
0430 20:02:45.335972 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:45.335991 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:45.336289 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:45.336577 (+ 288us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:45.338016 (+ 1439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.338903 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:45.340657 (+ 1754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.341056 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:45.344243 (+ 3187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:45.344289 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:45.344303 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:45.344345 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:45.346740 (+ 2395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.347323 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.347331 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.347338 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.347442 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.348063 (+ 621us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.348067 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.348068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.348553 (+ 485us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:45.348746 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:45.349230 (+ 484us) server_negotiation.cc:300] Negotiation successful
0430 20:02:45.349636 (+ 406us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":377,"threads_started":1}
I20260430 20:02:45.350399 19357 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:45.350736 19352 sys_catalog.cc:565] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:45.353258 19359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08cae824112450cb2fac043215188e0. Latest consensus state: current_term: 1 leader_uuid: "f08cae824112450cb2fac043215188e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } } }
I20260430 20:02:45.353351 19359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:45.353057 19358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08cae824112450cb2fac043215188e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } } }
I20260430 20:02:45.353572 19358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: This master's current role is: LEADER
W20260430 20:02:45.356166 19371 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:02:45.356282 19371 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:02:45.356360 19372 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:45.357169 19372 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:45.363054 19372 catalog_manager.cc:1357] Generated new cluster ID: efbe2f980bbf47c781dfd1f1d775dcf9
I20260430 20:02:45.363150 19372 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:45.380995 19372 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:02:45.381983 19372 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:45.388173 19372 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Generated new TSK 0
I20260430 20:02:45.388904 19372 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:45.460124 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41139
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:45.588090 19380 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:45.588330 19380 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:45.588392 19380 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:45.591830 19380 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:45.591923 19380 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:45.592008 19380 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:45.598809 19380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:45.600661 19380 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:45.602028 19380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:45.611857 19388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:45.611857 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:45.611877 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:45.612192 19380 server_base.cc:1061] running on GCE node
I20260430 20:02:45.612736 19380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:45.613391 19380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:45.614534 19380 hybrid_clock.cc:648] HybridClock initialized: now 1777579365614488 us; error 78 us; skew 500 ppm
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:45.618454 19380 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:02:45.620002 19380 webserver.cc:492] Webserver started at http://127.10.46.1:36411/ using document root <none> and password file <none>
I20260430 20:02:45.620851 19380 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:45.620951 19380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:45.621249 19380 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:45.623886 19380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "b7699349c19c4a14aa92e93339a49121"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "e09623d52e559a1644b95f931ba23d39"
server_key_iv: "b0441e010eee1fe97f5dcdac74ccc215"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.624598 19380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "b7699349c19c4a14aa92e93339a49121"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "e09623d52e559a1644b95f931ba23d39"
server_key_iv: "b0441e010eee1fe97f5dcdac74ccc215"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.629137 19380 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260430 20:02:45.631672 19395 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.633157 19380 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260430 20:02:45.633316 19380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "b7699349c19c4a14aa92e93339a49121"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "e09623d52e559a1644b95f931ba23d39"
server_key_iv: "b0441e010eee1fe97f5dcdac74ccc215"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.633445 19380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:45.648165 19380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:45.655689 19380 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:45.655998 19380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:45.656663 19380 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:45.657697 19380 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:45.657747 19380 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.657790 19380 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:45.657850 19380 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.670207 19380 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:43407
I20260430 20:02:45.670220 19508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:43407 every 8 connection(s)
I20260430 20:02:45.671224 19380 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:45.677987 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19380
I20260430 20:02:45.678148 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:02:45.678501 10424 external_mini_cluster.cc:1468] Setting key cabc09ff047fb03c6e9375b931881713
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:45.684511 19355 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:45.673206 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:52355 (local address 127.10.46.62:41139)
0430 20:02:45.673546 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:45.673550 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:45.673963 (+ 413us) server_negotiation.cc:408] Connection header received
0430 20:02:45.674856 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:45.674859 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:45.674909 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:45.674998 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:45.676647 (+ 1649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.677176 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:45.677913 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.678093 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:45.680518 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:45.680540 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:45.680547 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:45.680570 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:45.682169 (+ 1599us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.682845 (+ 676us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.682848 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.682849 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.682894 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.683270 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.683272 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.683274 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.683433 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:45.683516 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:45.684254 (+ 738us) server_negotiation.cc:300] Negotiation successful
0430 20:02:45.684369 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260430 20:02:45.685544 19511 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:45.673341 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:52355)
0430 20:02:45.673809 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:45.673849 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:45.674673 (+ 824us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:45.675146 (+ 473us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:45.675155 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:45.675549 (+ 394us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:45.676470 (+ 921us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:45.676483 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.677317 (+ 834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:45.677320 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:45.677766 (+ 446us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:45.677778 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.677998 (+ 220us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:45.678638 (+ 640us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:45.678663 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:45.680361 (+ 1698us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:45.682323 (+ 1962us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:45.682330 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:45.682346 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:45.682719 (+ 373us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:45.683001 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:45.683007 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:45.683012 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:45.683156 (+ 144us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:45.683549 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:45.683558 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:45.683901 (+ 343us) client_negotiation.cc:770] Sending connection context
0430 20:02:45.684175 (+ 274us) client_negotiation.cc:241] Negotiation successful
0430 20:02:45.684477 (+ 302us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":152,"threads_started":1}
I20260430 20:02:45.687448 19509 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41139
I20260430 20:02:45.687739 19509 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:45.688352 19509 heartbeater.cc:507] Master 127.10.46.62:41139 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:02:45.689996 19316 ts_manager.cc:194] Registered new tserver with Master: b7699349c19c4a14aa92e93339a49121 (127.10.46.1:43407)
I20260430 20:02:45.691139 19316 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:52355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:45.735661 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41139
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:45.842509 19516 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:45.842757 19516 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:45.842818 19516 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:45.846474 19516 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:45.846567 19516 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:45.846695 19516 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:45.851475 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:45.852864 19516 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:45.854115 19516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:45.861910 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:45.861948 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:45.862021 19516 server_base.cc:1061] running on GCE node
W20260430 20:02:45.862143 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:45.862676 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:45.863327 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:45.864629 19516 hybrid_clock.cc:648] HybridClock initialized: now 1777579365864468 us; error 160 us; skew 500 ppm
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:45.868117 19516 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:02:45.869342 19516 webserver.cc:492] Webserver started at http://127.10.46.2:36975/ using document root <none> and password file <none>
I20260430 20:02:45.869952 19516 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:45.870007 19516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:45.870224 19516 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:45.872110 19516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "79d0b731bb334dc8ac65e3ec190165bc"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "269e4527ba1b6126db9c2af6698f0273"
server_key_iv: "cf9dc7c58b2bd87620c83971d4f812f8"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.872637 19516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "79d0b731bb334dc8ac65e3ec190165bc"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "269e4527ba1b6126db9c2af6698f0273"
server_key_iv: "cf9dc7c58b2bd87620c83971d4f812f8"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.876689 19516 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:45.880113 19531 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.881359 19516 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:45.881531 19516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "79d0b731bb334dc8ac65e3ec190165bc"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "269e4527ba1b6126db9c2af6698f0273"
server_key_iv: "cf9dc7c58b2bd87620c83971d4f812f8"
server_key_version: "encryptionkey@0"
I20260430 20:02:45.881662 19516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:45.894100 19516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:45.897787 19516 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:45.898030 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:45.898687 19516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:45.899638 19516 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:45.899688 19516 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.899729 19516 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:45.899744 19516 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:45.911659 19516 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:37179
I20260430 20:02:45.911731 19644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:37179 every 8 connection(s)
I20260430 20:02:45.912680 19516 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:02:45.912936 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19516
I20260430 20:02:45.913086 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:02:45.913424 10424 external_mini_cluster.cc:1468] Setting key 0cb46f0d90314b0cf1b600dc43a52859
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Apr 30 20:02:45 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:45.931752 19355 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:45.914723 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:48379 (local address 127.10.46.62:41139)
0430 20:02:45.914908 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:45.914912 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:45.916245 (+ 1333us) server_negotiation.cc:408] Connection header received
0430 20:02:45.917310 (+ 1065us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:45.917315 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:45.917376 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:45.917499 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:45.919240 (+ 1741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.920017 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:45.920842 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.921023 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:45.927247 (+ 6224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:45.927275 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:45.927278 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:45.927312 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:45.928971 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.929727 (+ 756us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.929731 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.929732 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.929781 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:45.930241 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:45.930246 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:45.930248 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:45.930558 (+ 310us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:45.930665 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:45.931472 (+ 807us) server_negotiation.cc:300] Negotiation successful
0430 20:02:45.931600 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:02:45.932704 19647 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:45.915123 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:48379)
0430 20:02:45.916016 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:45.916082 (+ 66us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:45.917064 (+ 982us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:45.917664 (+ 600us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:45.917672 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:45.918075 (+ 403us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:45.919066 (+ 991us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:45.919082 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.920278 (+ 1196us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:45.920281 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:45.920707 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:45.920714 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:45.920967 (+ 253us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:45.921823 (+ 856us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:45.921855 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:45.926980 (+ 5125us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:45.929143 (+ 2163us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:45.929153 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:45.929169 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:45.929584 (+ 415us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:45.929895 (+ 311us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:45.929901 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:45.929906 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:45.930046 (+ 140us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:45.930699 (+ 653us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:45.930709 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:45.931053 (+ 344us) client_negotiation.cc:770] Sending connection context
0430 20:02:45.931316 (+ 263us) client_negotiation.cc:241] Negotiation successful
0430 20:02:45.931626 (+ 310us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":136,"threads_started":1}
I20260430 20:02:45.934206 19645 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41139
I20260430 20:02:45.934522 19645 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:45.935068 19645 heartbeater.cc:507] Master 127.10.46.62:41139 requested a full tablet report, sending...
I20260430 20:02:45.936424 19316 ts_manager.cc:194] Registered new tserver with Master: 79d0b731bb334dc8ac65e3ec190165bc (127.10.46.2:37179)
I20260430 20:02:45.937069 19316 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:48379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:02:45.976076 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:41139
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:46.086427 19652 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:46.086669 19652 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:46.086728 19652 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:46.090252 19652 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:46.090341 19652 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:46.090425 19652 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:46.094995 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:46.096261 19652 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:46.097085 19652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:46.104123 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.104226 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.104295 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:46.104589 19652 server_base.cc:1061] running on GCE node
I20260430 20:02:46.105036 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:46.105703 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:46.107013 19652 hybrid_clock.cc:648] HybridClock initialized: now 1777579366106939 us; error 81 us; skew 500 ppm
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:46.110846 19652 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:02:46.112290 19652 webserver.cc:492] Webserver started at http://127.10.46.3:36375/ using document root <none> and password file <none>
I20260430 20:02:46.112936 19652 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:46.113022 19652 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:46.113260 19652 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:02:46.115224 19652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "8182c06cad0946a4bfd3e809e8ba4169"
format_stamp: "Formatted at 2026-04-30 20:02:46 on dist-test-slave-0bdw"
server_key: "8d9a975ff2c67f94aafb598564f5cdec"
server_key_iv: "38ac779f1e385bc159b245afbfb9bcd0"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.115881 19652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "8182c06cad0946a4bfd3e809e8ba4169"
format_stamp: "Formatted at 2026-04-30 20:02:46 on dist-test-slave-0bdw"
server_key: "8d9a975ff2c67f94aafb598564f5cdec"
server_key_iv: "38ac779f1e385bc159b245afbfb9bcd0"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.120432 19652 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260430 20:02:46.122967 19667 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.124296 19652 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:02:46.124423 19652 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "8182c06cad0946a4bfd3e809e8ba4169"
format_stamp: "Formatted at 2026-04-30 20:02:46 on dist-test-slave-0bdw"
server_key: "8d9a975ff2c67f94aafb598564f5cdec"
server_key_iv: "38ac779f1e385bc159b245afbfb9bcd0"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.124527 19652 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:46.151400 19652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:46.154990 19652 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:46.155189 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:46.155733 19652 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:46.156878 19652 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:46.156936 19652 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.156980 19652 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:46.156996 19652 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.169063 19652 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:45727
I20260430 20:02:46.169076 19780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:45727 every 8 connection(s)
I20260430 20:02:46.170099 19652 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:46.173794 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19652
I20260430 20:02:46.173947 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:02:46.174237 10424 external_mini_cluster.cc:1468] Setting key a7b0bd75d8ec55be80d173af4edfe7c6
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:02:46.185930 19355 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.173005 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:39305 (local address 127.10.46.62:41139)
0430 20:02:46.173191 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.173195 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.173987 (+ 792us) server_negotiation.cc:408] Connection header received
0430 20:02:46.175542 (+ 1555us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.175547 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.175625 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.175750 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.177738 (+ 1988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.178248 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.178919 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.179090 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.181679 (+ 2589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:46.181708 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:46.181711 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:46.181737 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:46.183378 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:46.184317 (+ 939us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:46.184323 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:46.184325 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:46.184392 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:46.184780 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:46.184784 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:46.184788 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:46.184951 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:46.185064 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:46.185596 (+ 532us) server_negotiation.cc:300] Negotiation successful
0430 20:02:46.185735 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260430 20:02:46.186640 19783 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.173149 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:39305)
0430 20:02:46.173780 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.173830 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.175054 (+ 1224us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.175980 (+ 926us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.175990 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.176519 (+ 529us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.177570 (+ 1051us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.177587 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.178386 (+ 799us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.178396 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.178788 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.178795 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.179018 (+ 223us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.180003 (+ 985us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.180023 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.181511 (+ 1488us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:02:46.183704 (+ 2193us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:46.183713 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:46.183726 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:46.184153 (+ 427us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:46.184536 (+ 383us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:02:46.184539 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:02:46.184541 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:02:46.184651 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:02:46.185063 (+ 412us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:02:46.185070 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:02:46.185329 (+ 259us) client_negotiation.cc:770] Sending connection context
0430 20:02:46.185548 (+ 219us) client_negotiation.cc:241] Negotiation successful
0430 20:02:46.185792 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":122,"threads_started":1}
I20260430 20:02:46.187793 19781 heartbeater.cc:344] Connected to a master server at 127.10.46.62:41139
I20260430 20:02:46.188086 19781 heartbeater.cc:461] Registering TS with master...
I20260430 20:02:46.188629 19781 heartbeater.cc:507] Master 127.10.46.62:41139 requested a full tablet report, sending...
I20260430 20:02:46.189662 19316 ts_manager.cc:194] Registered new tserver with Master: 8182c06cad0946a4bfd3e809e8ba4169 (127.10.46.3:45727)
I20260430 20:02:46.190184 19316 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:39305
I20260430 20:02:46.202211 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.1.keytab.
Entry for principal oryx/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.1.keytab.
WARNING: no policy specified for oryx/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.2.keytab.
Entry for principal oryx/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.2.keytab.
WARNING: no policy specified for oryx/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.3.keytab.
Entry for principal oryx/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.3.keytab.
I20260430 20:02:46.291764 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19380
I20260430 20:02:46.298359 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19516
I20260430 20:02:46.304947 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19652
I20260430 20:02:46.311021 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19284
I20260430 20:02:46.318742 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41139
--webserver_interface=127.10.46.62
--webserver_port=44671
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:41139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:46.430763 19790 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:46.431047 19790 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:46.431108 19790 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:46.436206 19790 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:02:46.436302 19790 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:46.436327 19790 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:46.436347 19790 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:02:46.436365 19790 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:02:46.441563 19790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:41139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=44671
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:46.442806 19790 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:46.443697 19790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:46.449915 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.449895 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.450013 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:46.450502 19790 server_base.cc:1061] running on GCE node
I20260430 20:02:46.451054 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:46.452114 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:46.453308 19790 hybrid_clock.cc:648] HybridClock initialized: now 1777579366453249 us; error 75 us; skew 500 ppm
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:46.456377 19790 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:02:46.457584 19790 webserver.cc:492] Webserver started at http://127.10.46.62:44671/ using document root <none> and password file <none>
I20260430 20:02:46.458166 19790 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:46.458269 19790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:46.462144 19790 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260430 20:02:46.464865 19805 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.466076 19790 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260430 20:02:46.466259 19790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "f08cae824112450cb2fac043215188e0"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "6906fd845933808855314ffc1bf82c19"
server_key_iv: "ee573ca34a493738bf036e91c5d66ccf"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.466747 19790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:46.479755 19790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:46.482813 19790 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:46.483031 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:46.491314 19790 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:41139
I20260430 20:02:46.491313 19857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:41139 every 8 connection(s)
I20260430 20:02:46.492563 19790 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:02:46.495640 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19790
I20260430 20:02:46.496224 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:43407
--local_ip_for_outbound_sockets=127.10.46.1
--tserver_master_addrs=127.10.46.62:41139
--webserver_port=36411
--webserver_interface=127.10.46.1
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.46.1
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 20:02:46.497586 19858 sys_catalog.cc:263] Verifying existing consensus state
I20260430 20:02:46.499460 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Bootstrap starting.
I20260430 20:02:46.516127 19858 log.cc:826] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Log is configured to *not* fsync() on all Append() calls
I20260430 20:02:46.522578 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 20:02:46.523001 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Bootstrap complete.
I20260430 20:02:46.526556 19858 raft_consensus.cc:359] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:46.526880 19858 raft_consensus.cc:740] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08cae824112450cb2fac043215188e0, State: Initialized, Role: FOLLOWER
I20260430 20:02:46.527365 19858 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:46.527483 19858 raft_consensus.cc:399] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:02:46.527576 19858 raft_consensus.cc:493] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:02:46.527670 19858 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 1 FOLLOWER]: Advancing to term 2
I20260430 20:02:46.529307 19858 raft_consensus.cc:515] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:46.529671 19858 leader_election.cc:304] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08cae824112450cb2fac043215188e0; no voters:
I20260430 20:02:46.530005 19858 leader_election.cc:290] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260430 20:02:46.530104 19862 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 20:02:46.530501 19862 raft_consensus.cc:697] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [term 2 LEADER]: Becoming Leader. State: Replica: f08cae824112450cb2fac043215188e0, State: Running, Role: LEADER
I20260430 20:02:46.530799 19862 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } }
I20260430 20:02:46.531337 19858 sys_catalog.cc:565] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:02:46.531713 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f08cae824112450cb2fac043215188e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } } }
I20260430 20:02:46.531840 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:46.532021 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08cae824112450cb2fac043215188e0. Latest consensus state: current_term: 2 leader_uuid: "f08cae824112450cb2fac043215188e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cae824112450cb2fac043215188e0" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 41139 } } }
I20260430 20:02:46.532079 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0 [sys.catalog]: This master's current role is: LEADER
I20260430 20:02:46.534229 19866 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:02:46.537205 19866 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:02:46.537873 19866 catalog_manager.cc:1269] Loaded cluster ID: efbe2f980bbf47c781dfd1f1d775dcf9
I20260430 20:02:46.537940 19866 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:02:46.540282 19866 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:02:46.541023 19866 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f08cae824112450cb2fac043215188e0: Loaded TSK: 0
I20260430 20:02:46.542013 19866 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260430 20:02:46.609058 19860 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:46.609391 19860 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:46.609493 19860 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:46.613317 19860 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:46.613438 19860 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:46.613555 19860 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:02:46.618297 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.1.keytab
--principal=oryx/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:43407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=36411
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:46.619504 19860 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:46.620496 19860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:46.627722 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.627835 19885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.627880 19887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:46.628253 19860 server_base.cc:1061] running on GCE node
I20260430 20:02:46.628716 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:46.629433 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:46.630740 19860 hybrid_clock.cc:648] HybridClock initialized: now 1777579366630710 us; error 67 us; skew 500 ppm
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:46.634979 19860 init.cc:377] Logged in from keytab as oryx/127.10.46.1@KRBTEST.COM (short username oryx)
I20260430 20:02:46.636539 19860 webserver.cc:492] Webserver started at http://127.10.46.1:36411/ using document root <none> and password file <none>
I20260430 20:02:46.637292 19860 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:46.637385 19860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:46.642371 19860 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:02:46.645632 19894 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.647403 19860 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260430 20:02:46.647591 19860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "b7699349c19c4a14aa92e93339a49121"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "e09623d52e559a1644b95f931ba23d39"
server_key_iv: "b0441e010eee1fe97f5dcdac74ccc215"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.648208 19860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:46.662886 19860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:46.667642 19860 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:46.668105 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:46.668963 19860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:46.670197 19860 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:46.670285 19860 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.670377 19860 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:46.670420 19860 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.683323 19860 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:43407
I20260430 20:02:46.683337 20007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:43407 every 8 connection(s)
I20260430 20:02:46.684866 19860 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:02:46.693475 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 19860
I20260430 20:02:46.693998 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:37179
--local_ip_for_outbound_sockets=127.10.46.2
--tserver_master_addrs=127.10.46.62:41139
--webserver_port=36975
--webserver_interface=127.10.46.2
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.46.2
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.703903 20010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.687846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:48189)
0430 20:02:46.688390 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.688435 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.689322 (+ 887us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.690952 (+ 1630us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.690965 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.691414 (+ 449us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.692975 (+ 1561us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.693115 (+ 140us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.696322 (+ 3207us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.696330 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.697471 (+ 1141us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.697494 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.698146 (+ 652us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.699616 (+ 1470us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.699640 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.702857 (+ 3217us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
W20260430 20:02:46.704375 20008 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:41139 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.46.62:41139: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
W20260430 20:02:46.706063 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.689284 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:48189 (local address 127.10.46.62:41139)
0430 20:02:46.689778 (+ 494us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.689789 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.689830 (+ 41us) server_negotiation.cc:408] Connection header received
0430 20:02:46.690125 (+ 295us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.690141 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.690512 (+ 371us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.690879 (+ 367us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.693629 (+ 2750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.695991 (+ 2362us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.698691 (+ 2700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.699215 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.705183 (+ 5968us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:48189: BlockingRecv error: recv got EOF from 127.10.46.1:48189 (error 108)
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":136,"threads_started":1}
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.713526 20010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.704864 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:48367)
0430 20:02:46.706780 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.706799 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.706933 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.707419 (+ 486us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.707424 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.707649 (+ 225us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.707974 (+ 325us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.707985 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.709194 (+ 1209us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.709200 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.709844 (+ 644us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.709857 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.709991 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.710680 (+ 689us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.710702 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.713321 (+ 2619us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1825,"thread_start_us":107,"threads_started":1}
W20260430 20:02:46.713971 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.704947 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:48367 (local address 127.10.46.62:41139)
0430 20:02:46.706447 (+ 1500us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.706452 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.707040 (+ 588us) server_negotiation.cc:408] Connection header received
0430 20:02:46.707098 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.707103 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.707170 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.707271 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.708142 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.709034 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.710268 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.710504 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.713849 (+ 3345us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:48367: BlockingRecv error: recv got EOF from 127.10.46.1:48367 (error 108)
Metrics: {"server-negotiator.queue_time_us":1397,"thread_start_us":133,"threads_started":1}
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.720872 20010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.714124 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:49557)
0430 20:02:46.714319 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.714336 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.714433 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.714837 (+ 404us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.714842 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.715040 (+ 198us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.715329 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.715340 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.716520 (+ 1180us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.716527 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.717160 (+ 633us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.717171 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.717319 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.717934 (+ 615us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.717955 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.720709 (+ 2754us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260430 20:02:46.721472 20008 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260430 20:02:46.720944 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.714251 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:49557 (local address 127.10.46.62:41139)
0430 20:02:46.714393 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.714397 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.714413 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:02:46.714565 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.714570 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.714634 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.714714 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.715488 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.716368 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.717574 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.717788 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.720832 (+ 3044us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:49557: BlockingRecv error: recv got EOF from 127.10.46.1:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260430 20:02:46.823900 20012 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:46.824138 20012 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:46.824246 20012 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:46.828006 20012 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:46.828086 20012 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:46.828173 20012 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:02:46.832799 20012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.2.keytab
--principal=oryx/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:37179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=36975
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:46.834033 20012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:46.834941 20012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 20:02:46.842486 20012 server_base.cc:1061] running on GCE node
W20260430 20:02:46.842137 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.842137 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:46.842554 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:46.843212 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:46.843933 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:46.845203 20012 hybrid_clock.cc:648] HybridClock initialized: now 1777579366845148 us; error 57 us; skew 500 ppm
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:46.848443 20012 init.cc:377] Logged in from keytab as oryx/127.10.46.2@KRBTEST.COM (short username oryx)
I20260430 20:02:46.849633 20012 webserver.cc:492] Webserver started at http://127.10.46.2:36975/ using document root <none> and password file <none>
I20260430 20:02:46.850440 20012 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:46.850651 20012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:46.854846 20012 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:46.857209 20029 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.858585 20012 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 20:02:46.858779 20012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "79d0b731bb334dc8ac65e3ec190165bc"
format_stamp: "Formatted at 2026-04-30 20:02:45 on dist-test-slave-0bdw"
server_key: "269e4527ba1b6126db9c2af6698f0273"
server_key_iv: "cf9dc7c58b2bd87620c83971d4f812f8"
server_key_version: "encryptionkey@0"
I20260430 20:02:46.859289 20012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:46.875046 20012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:46.878739 20012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:46.878978 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:46.879595 20012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:46.880620 20012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:46.880698 20012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.880769 20012 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:46.880821 20012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:46.892117 20012 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:37179
I20260430 20:02:46.892131 20142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:37179 every 8 connection(s)
I20260430 20:02:46.893267 20012 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.902900 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20012
I20260430 20:02:46.903465 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:45727
--local_ip_for_outbound_sockets=127.10.46.3
--tserver_master_addrs=127.10.46.62:41139
--webserver_port=36375
--webserver_interface=127.10.46.3
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.46.3
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:02:46.903915 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.895320 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:34979 (local address 127.10.46.62:41139)
0430 20:02:46.895521 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.895525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.896204 (+ 679us) server_negotiation.cc:408] Connection header received
0430 20:02:46.897107 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.897111 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.897181 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.897291 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.898902 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.899398 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.900149 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.900325 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.903738 (+ 3413us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:34979: BlockingRecv error: recv got EOF from 127.10.46.2:34979 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260430 20:02:46.904430 20145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.895578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:34979)
0430 20:02:46.896041 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.896074 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.896903 (+ 829us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.897460 (+ 557us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.897468 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.897851 (+ 383us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.898711 (+ 860us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.898725 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.899539 (+ 814us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.899544 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.900017 (+ 473us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.900025 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.900253 (+ 228us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.900911 (+ 658us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.900930 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.903466 (+ 2536us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
W20260430 20:02:46.905028 20143 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:41139 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.46.62:41139: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.911139 20145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.905631 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:44127)
0430 20:02:46.905861 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.905881 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.905994 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.906300 (+ 306us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.906304 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.906488 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.906690 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.906695 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.907452 (+ 757us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.907456 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.907925 (+ 469us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.907934 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.908064 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.908729 (+ 665us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.908743 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.910992 (+ 2249us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260430 20:02:46.911274 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.905716 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:44127 (local address 127.10.46.62:41139)
0430 20:02:46.905852 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.905856 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.905949 (+ 93us) server_negotiation.cc:408] Connection header received
0430 20:02:46.906125 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.906129 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.906190 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.906281 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.906822 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.907313 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.908046 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.908246 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.911158 (+ 2912us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:44127: BlockingRecv error: recv got EOF from 127.10.46.2:44127 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:46 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:46.917309 20145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:46.911763 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:50039)
0430 20:02:46.912004 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:46.912016 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:46.912088 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:46.912409 (+ 321us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:46.912412 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:46.912555 (+ 143us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:46.912726 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.912730 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.913837 (+ 1107us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.913841 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:46.914288 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:46.914295 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.914405 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.914971 (+ 566us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:46.914986 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:46.917157 (+ 2171us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260430 20:02:46.917497 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:46.911898 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:50039 (local address 127.10.46.62:41139)
0430 20:02:46.912061 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:46.912068 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:46.912088 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:02:46.912191 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:46.912197 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:46.912261 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:46.912362 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:46.912855 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.913679 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:46.914412 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:46.914625 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:46.917336 (+ 2711us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:50039: BlockingRecv error: recv got EOF from 127.10.46.2:50039 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260430 20:02:46.917639 20143 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260430 20:02:47.015208 20146 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:02:47.015458 20146 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:02:47.015520 20146 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:02:47.018882 20146 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:02:47.018958 20146 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:02:47.019042 20146 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:02:47.023672 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/krb5kdc/oryx_127.10.46.3.keytab
--principal=oryx/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:45727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=36375
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:02:47.024852 20146 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:02:47.025681 20146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:02:47.032750 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:02:47.032817 20152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:47.033015 20146 server_base.cc:1061] running on GCE node
W20260430 20:02:47.032750 20154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:02:47.033447 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:02:47.034157 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:02:47.035336 20146 hybrid_clock.cc:648] HybridClock initialized: now 1777579367035299 us; error 42 us; skew 500 ppm
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:02:47.038368 20146 init.cc:377] Logged in from keytab as oryx/127.10.46.3@KRBTEST.COM (short username oryx)
I20260430 20:02:47.039529 20146 webserver.cc:492] Webserver started at http://127.10.46.3:36375/ using document root <none> and password file <none>
I20260430 20:02:47.040174 20146 fs_manager.cc:362] Metadata directory not provided
I20260430 20:02:47.040280 20146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:02:47.044034 20146 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260430 20:02:47.046098 20161 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:47.047138 20146 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:02:47.047298 20146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "8182c06cad0946a4bfd3e809e8ba4169"
format_stamp: "Formatted at 2026-04-30 20:02:46 on dist-test-slave-0bdw"
server_key: "8d9a975ff2c67f94aafb598564f5cdec"
server_key_iv: "38ac779f1e385bc159b245afbfb9bcd0"
server_key_version: "encryptionkey@0"
I20260430 20:02:47.047763 20146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:02:47.064350 20146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:02:47.068223 20146 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:02:47.068459 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:02:47.069061 20146 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:02:47.069996 20146 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:02:47.070070 20146 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:47.070143 20146 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:02:47.070199 20146 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:02:47.081565 20146 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:45727
I20260430 20:02:47.081636 20274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:45727 every 8 connection(s)
I20260430 20:02:47.082718 20146 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:02:47.090740 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20146
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
W20260430 20:02:47.094017 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:47.084963 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:51597 (local address 127.10.46.62:41139)
0430 20:02:47.085152 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.085157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.085954 (+ 797us) server_negotiation.cc:408] Connection header received
0430 20:02:47.086795 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.086798 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.086851 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.086933 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.088744 (+ 1811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.089269 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.089958 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.090103 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.093907 (+ 3804us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:51597: BlockingRecv error: recv got EOF from 127.10.46.3:51597 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 20:02:47.094558 20277 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.085329 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:51597)
0430 20:02:47.085795 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:47.085831 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:47.086594 (+ 763us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:47.087151 (+ 557us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:47.087159 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:47.087572 (+ 413us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:47.088557 (+ 985us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.088571 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.089415 (+ 844us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.089418 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:47.089821 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.089828 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.090001 (+ 173us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.090756 (+ 755us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:47.090775 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:47.093761 (+ 2986us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":114,"threads_started":1}
W20260430 20:02:47.095098 20275 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:41139 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.46.62:41139: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:47.101164 20277 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.095614 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:51873)
0430 20:02:47.095826 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:47.095842 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:47.095926 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:47.096182 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:47.096189 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:47.096373 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:47.096635 (+ 262us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.096644 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.097632 (+ 988us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.097635 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:47.098142 (+ 507us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.098149 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.098239 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.098650 (+ 411us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:47.098663 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:47.101014 (+ 2351us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260430 20:02:47.101399 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:47.095722 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:51873 (local address 127.10.46.62:41139)
0430 20:02:47.095877 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.095881 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.095894 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:02:47.096019 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.096022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.096063 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.096166 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.096785 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.097493 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.098261 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.098489 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.101285 (+ 2796us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:51873: BlockingRecv error: recv got EOF from 127.10.46.3:51873 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260430 20:02:47.102873 20014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.091814 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.10.46.62:41139)
0430 20:02:47.091980 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.091986 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.092001 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:47.092041 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.092047 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.092098 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.092200 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.093255 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.094113 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.095026 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.095301 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.096820 (+ 1519us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:02:47.096842 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:02:47.096859 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:02:47.096893 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:02:47.099023 (+ 2130us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:47.099498 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:47.099505 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:47.099514 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:47.099624 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:02:47.099947 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:02:47.099949 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:02:47.099950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:02:47.100455 (+ 505us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:02:47.100667 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0430 20:02:47.101247 (+ 580us) server_negotiation.cc:300] Negotiation successful
0430 20:02:47.102558 (+ 1311us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:47.106855 20277 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.101771 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:33137)
0430 20:02:47.102014 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:47.102026 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:47.102100 (+ 74us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:47.102370 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:47.102373 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:47.102499 (+ 126us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:47.102656 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.102660 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.103491 (+ 831us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.103495 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:47.104006 (+ 511us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.104014 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.104103 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.104614 (+ 511us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:47.104633 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:47.106710 (+ 2077us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260430 20:02:47.106916 20011 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:47.101810 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:33137 (local address 127.10.46.62:41139)
0430 20:02:47.101941 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.101944 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.102108 (+ 164us) server_negotiation.cc:408] Connection header received
0430 20:02:47.102164 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.102169 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.102247 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.102340 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.102788 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.103363 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.104114 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.104294 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.106816 (+ 2522us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:33137: BlockingRecv error: recv got EOF from 127.10.46.3:33137 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260430 20:02:47.107121 20275 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:47.727818 20279 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.722215 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:50969)
0430 20:02:47.722506 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:47.722521 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:47.722622 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:47.722903 (+ 281us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:47.722905 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:47.723086 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:47.723355 (+ 269us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.723361 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.724178 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.724181 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:47.724614 (+ 433us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.724621 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.724723 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.725507 (+ 784us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:47.725521 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:47.727601 (+ 2080us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
W20260430 20:02:47.727909 20280 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:47.722318 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50969 (local address 127.10.46.62:41139)
0430 20:02:47.722600 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.722603 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.722620 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:47.722727 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.722730 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.722776 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.722864 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.723499 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.724059 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.724749 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.724896 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.727736 (+ 2840us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:50969: BlockingRecv error: recv got EOF from 127.10.46.1:50969 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:47 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:47.923635 20281 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:47.918348 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:47229)
0430 20:02:47.918720 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:47.918733 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:47.918841 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:47.919154 (+ 313us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:47.919157 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:47.919349 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:47.919592 (+ 243us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.919598 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.920466 (+ 868us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.920470 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:47.920866 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:47.920880 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.920977 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.921537 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:47.921550 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:47.923495 (+ 1945us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":90,"threads_started":1}
W20260430 20:02:47.923820 20280 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:47.918493 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:47229 (local address 127.10.46.62:41139)
0430 20:02:47.918670 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:47.918674 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:47.918838 (+ 164us) server_negotiation.cc:408] Connection header received
0430 20:02:47.918935 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:47.918937 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:47.918992 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:47.919062 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:47.919752 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.920348 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:47.920988 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:47.921160 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:47.923657 (+ 2497us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:47229: BlockingRecv error: recv got EOF from 127.10.46.2:47229 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:48.114099 20282 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:48.107938 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:42261)
0430 20:02:48.108382 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:48.108397 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:48.108565 (+ 168us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:48.108875 (+ 310us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:48.108879 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:48.109104 (+ 225us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:48.109377 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.109388 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.110401 (+ 1013us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.110405 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:48.110920 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.110930 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.111060 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.111743 (+ 683us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:48.111758 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:48.113958 (+ 2200us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":100,"threads_started":1}
W20260430 20:02:48.114326 20280 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:48.108097 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:42261 (local address 127.10.46.62:41139)
0430 20:02:48.108312 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:48.108317 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:48.108491 (+ 174us) server_negotiation.cc:408] Connection header received
0430 20:02:48.108694 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:48.108703 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:48.108755 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:48.108851 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:48.109558 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.110252 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.111109 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.111359 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.114134 (+ 2775us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:42261: BlockingRecv error: recv got EOF from 127.10.46.3:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:48.734565 20283 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:48.728903 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:42671)
0430 20:02:48.729201 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:48.729216 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:48.729320 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:48.729677 (+ 357us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:48.729680 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:48.729859 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:48.730076 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.730081 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.730872 (+ 791us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.730875 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:48.731284 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.731291 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.731402 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.732018 (+ 616us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:48.732040 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:48.734389 (+ 2349us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
W20260430 20:02:48.734673 20284 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:48.728990 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:42671 (local address 127.10.46.62:41139)
0430 20:02:48.729236 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:48.729240 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:48.729274 (+ 34us) server_negotiation.cc:408] Connection header received
0430 20:02:48.729477 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:48.729480 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:48.729530 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:48.729617 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:48.730219 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.730749 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.731402 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.731563 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.734540 (+ 2977us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:42671: BlockingRecv error: recv got EOF from 127.10.46.1:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:48 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:48.930299 20285 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:48.924623 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:47465)
0430 20:02:48.925119 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:48.925145 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:48.925496 (+ 351us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:48.925811 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:48.925815 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:48.926034 (+ 219us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:48.926233 (+ 199us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.926239 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.926988 (+ 749us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.926993 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:48.927394 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:48.927400 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.927533 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.928143 (+ 610us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:48.928159 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:48.930125 (+ 1966us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":118,"threads_started":1}
W20260430 20:02:48.930405 20284 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:48.924769 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:47465 (local address 127.10.46.62:41139)
0430 20:02:48.925069 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:48.925075 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:48.925388 (+ 313us) server_negotiation.cc:408] Connection header received
0430 20:02:48.925626 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:48.925630 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:48.925693 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:48.925788 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:48.926367 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.926868 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:48.927515 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:48.927717 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:48.930286 (+ 2569us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:47465: BlockingRecv error: recv got EOF from 127.10.46.2:47465 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:49.119989 20286 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:49.115103 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:46851)
0430 20:02:49.115408 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:49.115422 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:49.115582 (+ 160us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:49.115855 (+ 273us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:49.115857 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:49.116039 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:49.116242 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.116247 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.116927 (+ 680us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.116929 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:49.117320 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.117326 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.117416 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.117976 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:49.117989 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:49.119861 (+ 1872us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
W20260430 20:02:49.120103 20284 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:49.115282 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:46851 (local address 127.10.46.62:41139)
0430 20:02:49.115429 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:49.115433 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:49.115502 (+ 69us) server_negotiation.cc:408] Connection header received
0430 20:02:49.115679 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:49.115682 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:49.115726 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:49.115824 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:49.116345 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.116829 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.117439 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.117583 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.119973 (+ 2390us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:46851: BlockingRecv error: recv got EOF from 127.10.46.3:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:49.740463 20287 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:49.735476 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:36929)
0430 20:02:49.735795 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:49.735810 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:49.735903 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:49.736159 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:49.736162 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:49.736346 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:49.736561 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.736566 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.737352 (+ 786us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.737355 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:49.737760 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.737766 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.737858 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.738465 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:49.738478 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:49.740334 (+ 1856us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
W20260430 20:02:49.740657 20288 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:49.735621 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:36929 (local address 127.10.46.62:41139)
0430 20:02:49.735898 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:49.735902 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:49.735917 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:49.735978 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:49.735981 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:49.736027 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:49.736118 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:49.736695 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.737234 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.737885 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.738046 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.740498 (+ 2452us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:36929: BlockingRecv error: recv got EOF from 127.10.46.1:36929 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:49 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:49.937749 20289 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:49.931314 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:54943)
0430 20:02:49.931740 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:49.931758 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:49.931938 (+ 180us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:49.932237 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:49.932240 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:49.932479 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:49.932676 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.932682 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.933954 (+ 1272us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.933967 (+ 13us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:49.934418 (+ 451us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:49.934429 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.934623 (+ 194us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.935214 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:49.935229 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:49.937561 (+ 2332us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":124,"threads_started":1}
W20260430 20:02:49.937793 20288 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:49.931502 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:54943 (local address 127.10.46.62:41139)
0430 20:02:49.931710 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:49.931715 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:49.931871 (+ 156us) server_negotiation.cc:408] Connection header received
0430 20:02:49.932050 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:49.932055 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:49.932111 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:49.932205 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:49.932811 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.933515 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:49.934600 (+ 1085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:49.934834 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:49.937686 (+ 2852us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:54943: BlockingRecv error: recv got EOF from 127.10.46.2:54943 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:50.126356 20290 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:50.121009 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:57985)
0430 20:02:50.121304 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:50.121318 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:50.121436 (+ 118us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:50.121818 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:50.121821 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:50.122005 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:50.122227 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.122233 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.123000 (+ 767us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.123004 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:50.123456 (+ 452us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.123465 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.123589 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.124225 (+ 636us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:50.124240 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:50.126218 (+ 1978us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
W20260430 20:02:50.126567 20288 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:50.121183 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:57985 (local address 127.10.46.62:41139)
0430 20:02:50.121314 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:50.121317 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:50.121457 (+ 140us) server_negotiation.cc:408] Connection header received
0430 20:02:50.121533 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:50.121536 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:50.121617 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:50.121721 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:50.122352 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.122848 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.123607 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.123760 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.126383 (+ 2623us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:57985: BlockingRecv error: recv got EOF from 127.10.46.3:57985 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:50.751251 20291 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:50.741356 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:47089)
0430 20:02:50.741673 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:50.741688 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:50.741781 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:50.742053 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:50.742056 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:50.742258 (+ 202us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:50.742459 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.742464 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.743335 (+ 871us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.743339 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:50.743741 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.743747 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.743899 (+ 152us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.749130 (+ 5231us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:50.749148 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:50.751059 (+ 1911us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":137,"threads_started":1}
W20260430 20:02:50.751334 20292 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:50.741520 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:47089 (local address 127.10.46.62:41139)
0430 20:02:50.741815 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:50.741818 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:50.741833 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:50.741878 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:50.741881 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:50.741927 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:50.742020 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:50.742615 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.743197 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.743911 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.744117 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.751211 (+ 7094us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:47089: BlockingRecv error: recv got EOF from 127.10.46.1:47089 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:50 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:50.943547 20293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:50.938649 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:45267)
0430 20:02:50.938937 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:50.938951 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:50.939064 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:50.939326 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:50.939329 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:50.939522 (+ 193us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:50.939715 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.939721 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.940532 (+ 811us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.940536 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:50.940918 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:50.940924 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.941020 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.941608 (+ 588us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:50.941622 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:50.943423 (+ 1801us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":117,"threads_started":1}
W20260430 20:02:50.943764 20292 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:50.938717 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:45267 (local address 127.10.46.62:41139)
0430 20:02:50.938875 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:50.938880 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:50.939049 (+ 169us) server_negotiation.cc:408] Connection header received
0430 20:02:50.939136 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:50.939138 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:50.939216 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:50.939290 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:50.939901 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.940420 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:50.941019 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:50.941149 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:50.943630 (+ 2481us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:45267: BlockingRecv error: recv got EOF from 127.10.46.2:45267 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:51.132686 20294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:51.127330 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:54701)
0430 20:02:51.127716 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:51.127733 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:51.127883 (+ 150us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:51.128176 (+ 293us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:51.128179 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:51.128444 (+ 265us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:51.128740 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.128748 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.129677 (+ 929us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.129680 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:51.130061 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.130067 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.130163 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.130742 (+ 579us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:51.130755 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:51.132552 (+ 1797us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":105,"threads_started":1}
W20260430 20:02:51.132740 20292 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:51.127498 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:54701 (local address 127.10.46.62:41139)
0430 20:02:51.127685 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:51.127689 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:51.127816 (+ 127us) server_negotiation.cc:408] Connection header received
0430 20:02:51.127976 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:51.127981 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:51.128041 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:51.128120 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:51.128904 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.129553 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.130173 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.130297 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.132648 (+ 2351us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:54701: BlockingRecv error: recv got EOF from 127.10.46.3:54701 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:51.759068 20295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:51.752323 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:50027)
0430 20:02:51.752675 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:51.752691 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:51.752802 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:51.753079 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:51.753082 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:51.753298 (+ 216us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:51.753501 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.753506 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.754603 (+ 1097us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.754610 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:51.755115 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.755123 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.755220 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.755822 (+ 602us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:51.755835 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:51.758877 (+ 3042us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
W20260430 20:02:51.759151 20296 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:51.752427 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50027 (local address 127.10.46.62:41139)
0430 20:02:51.752683 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:51.752686 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:51.752798 (+ 112us) server_negotiation.cc:408] Connection header received
0430 20:02:51.752881 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:51.752884 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:51.752944 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:51.753032 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:51.753652 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.754419 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.755243 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.755391 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.759021 (+ 3630us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:50027: BlockingRecv error: recv got EOF from 127.10.46.1:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:51 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:51.951089 20297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:51.944726 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:43305)
0430 20:02:51.945137 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:51.945159 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:51.945340 (+ 181us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:51.946002 (+ 662us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:51.946008 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:51.946234 (+ 226us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:51.946462 (+ 228us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.946469 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.947655 (+ 1186us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.947662 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:51.948323 (+ 661us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:51.948335 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.948497 (+ 162us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.949112 (+ 615us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:51.949130 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:51.950957 (+ 1827us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
W20260430 20:02:51.951689 20296 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:51.944988 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:43305 (local address 127.10.46.62:41139)
0430 20:02:51.945243 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:51.945248 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:51.945268 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:02:51.945650 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:51.945659 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:51.945756 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:51.945907 (+ 151us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:51.946633 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.947456 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:51.948526 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:51.948762 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:51.951593 (+ 2831us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:43305: BlockingRecv error: recv got EOF from 127.10.46.2:43305 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:52.139034 20298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:52.133592 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:50469)
0430 20:02:52.133888 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:52.133902 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:52.134000 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:52.134305 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:52.134308 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:52.134518 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:52.134713 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.134717 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.135643 (+ 926us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.135646 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:52.136076 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.136083 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.136217 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.136815 (+ 598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:52.136829 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:52.138881 (+ 2052us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
W20260430 20:02:52.139259 20296 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:52.133733 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:50469 (local address 127.10.46.62:41139)
0430 20:02:52.133894 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:52.133897 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:52.134003 (+ 106us) server_negotiation.cc:408] Connection header received
0430 20:02:52.134069 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:52.134071 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:52.134150 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:52.134230 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:52.134880 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.135480 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.136255 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.136404 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.139028 (+ 2624us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:50469: BlockingRecv error: recv got EOF from 127.10.46.3:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:52.765796 20299 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:52.760028 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:59663)
0430 20:02:52.760330 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:52.760344 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:52.760453 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:52.760769 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:52.760771 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:52.760986 (+ 215us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:52.761247 (+ 261us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.761254 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.762358 (+ 1104us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.762361 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:52.762791 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.762798 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.762893 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.763567 (+ 674us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:52.763581 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:52.765619 (+ 2038us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
W20260430 20:02:52.765868 20300 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:52.760141 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:59663 (local address 127.10.46.62:41139)
0430 20:02:52.760375 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:52.760379 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:52.760397 (+ 18us) server_negotiation.cc:408] Connection header received
0430 20:02:52.760593 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:52.760597 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:52.760649 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:52.760736 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:52.761476 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.762253 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.762933 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.763184 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.765749 (+ 2565us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:59663: BlockingRecv error: recv got EOF from 127.10.46.1:59663 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:52 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:52.957867 20301 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:52.952506 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:46935)
0430 20:02:52.952778 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:52.952793 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:52.952935 (+ 142us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:52.953248 (+ 313us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:52.953251 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:52.953471 (+ 220us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:52.953656 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.953661 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.954420 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.954425 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:52.954806 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:52.954811 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.954940 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.955500 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:52.955514 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:52.957745 (+ 2231us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
W20260430 20:02:52.958055 20300 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:52.952572 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:46935 (local address 127.10.46.62:41139)
0430 20:02:52.952722 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:52.952727 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:52.952883 (+ 156us) server_negotiation.cc:408] Connection header received
0430 20:02:52.953043 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:52.953046 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:52.953134 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:52.953226 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:52.953783 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.954301 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:52.954921 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:52.955167 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:52.957918 (+ 2751us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:46935: BlockingRecv error: recv got EOF from 127.10.46.2:46935 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:53.145965 20302 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:53.140055 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:48711)
0430 20:02:53.140356 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:53.140371 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:53.140477 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:53.140797 (+ 320us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:53.140801 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:53.140981 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:53.141262 (+ 281us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.141268 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.142118 (+ 850us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.142122 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:53.142530 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.142537 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.142631 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.143492 (+ 861us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:53.143512 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:53.145843 (+ 2331us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
W20260430 20:02:53.146183 20300 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:53.140279 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:48711 (local address 127.10.46.62:41139)
0430 20:02:53.140420 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:53.140423 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:53.140438 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:53.140586 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:53.140589 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:53.140637 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:53.140722 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:53.141401 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.141897 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.142673 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.142928 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.146033 (+ 3105us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:48711: BlockingRecv error: recv got EOF from 127.10.46.3:48711 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:53.772642 20303 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:53.766797 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:35647)
0430 20:02:53.767234 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:53.767335 (+ 101us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:53.767461 (+ 126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:53.767723 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:53.767726 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:53.767944 (+ 218us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:53.768180 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.768186 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.769358 (+ 1172us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.769361 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:53.769746 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.769752 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.769843 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.770438 (+ 595us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:53.770451 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:53.772526 (+ 2075us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":98,"threads_started":1}
W20260430 20:02:53.772852 20304 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:53.766976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:35647 (local address 127.10.46.62:41139)
0430 20:02:53.767290 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:53.767294 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:53.767489 (+ 195us) server_negotiation.cc:408] Connection header received
0430 20:02:53.767539 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:53.767541 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:53.767592 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:53.767681 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:53.768339 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.769230 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.770048 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.770216 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.772713 (+ 2497us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:35647: BlockingRecv error: recv got EOF from 127.10.46.1:35647 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":74,"threads_started":1}
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:53 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:53.964001 20305 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:53.958846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:38591)
0430 20:02:53.959230 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:53.959243 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:53.959381 (+ 138us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:53.959637 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:53.959640 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:53.959870 (+ 230us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:53.960060 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.960065 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.960813 (+ 748us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.960819 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:53.961210 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:53.961216 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.961346 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.961926 (+ 580us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:53.961939 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:53.963884 (+ 1945us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":105,"threads_started":1}
W20260430 20:02:53.964162 20304 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:53.958915 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:38591 (local address 127.10.46.62:41139)
0430 20:02:53.959076 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:53.959080 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:53.959321 (+ 241us) server_negotiation.cc:408] Connection header received
0430 20:02:53.959471 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:53.959474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:53.959529 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:53.959617 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:53.960191 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.960696 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:53.961329 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:53.961534 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:53.964028 (+ 2494us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:38591: BlockingRecv error: recv got EOF from 127.10.46.2:38591 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:54.152697 20306 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:54.146892 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:41169)
0430 20:02:54.147219 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:54.147233 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:54.147339 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:54.147653 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:54.147656 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:54.147862 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:54.148066 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.148071 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.149188 (+ 1117us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.149193 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:54.149598 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.149606 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.149732 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.150357 (+ 625us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:54.150371 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:54.152571 (+ 2200us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
W20260430 20:02:54.152915 20304 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:54.147063 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:41169 (local address 127.10.46.62:41139)
0430 20:02:54.147223 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:54.147226 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:54.147308 (+ 82us) server_negotiation.cc:408] Connection header received
0430 20:02:54.147445 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:54.147448 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:54.147498 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:54.147587 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:54.148282 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.149023 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.149727 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.149936 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.152784 (+ 2848us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:41169: BlockingRecv error: recv got EOF from 127.10.46.3:41169 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:54.778966 20307 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:54.773519 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:36239)
0430 20:02:54.773823 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:54.773840 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:54.773948 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:54.774263 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:54.774266 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:54.774481 (+ 215us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:54.774705 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.774711 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.775504 (+ 793us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.775515 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:54.775920 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.775926 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.776060 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.776754 (+ 694us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:54.776768 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:54.778780 (+ 2012us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
W20260430 20:02:54.779039 20308 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:54.773617 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:36239 (local address 127.10.46.62:41139)
0430 20:02:54.773870 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:54.773876 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:54.773891 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:02:54.774064 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:54.774067 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:54.774148 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:54.774243 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:54.774835 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.775375 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.776041 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.776260 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.778921 (+ 2661us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:36239: BlockingRecv error: recv got EOF from 127.10.46.1:36239 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:54 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:54.970300 20309 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:54.964941 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:59491)
0430 20:02:54.965262 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:54.965281 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:54.965425 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:54.965735 (+ 310us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:54.965738 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:54.965918 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:54.966118 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.966123 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.967088 (+ 965us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.967092 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:54.967509 (+ 417us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:54.967515 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.967619 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.968241 (+ 622us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:54.968255 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:54.970177 (+ 1922us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":167,"threads_started":1}
W20260430 20:02:54.970649 20308 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:54.965052 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:59491 (local address 127.10.46.62:41139)
0430 20:02:54.965206 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:54.965210 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:54.965408 (+ 198us) server_negotiation.cc:408] Connection header received
0430 20:02:54.965511 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:54.965514 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:54.965564 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:54.965695 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:54.966267 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.966956 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:54.967950 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:54.968228 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:54.970526 (+ 2298us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:59491: BlockingRecv error: recv got EOF from 127.10.46.2:59491 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:55.159410 20310 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:55.153530 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:41265)
0430 20:02:55.153841 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:55.153860 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:55.153973 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:55.154484 (+ 511us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:55.154487 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:55.154721 (+ 234us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:55.155068 (+ 347us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.155078 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.156347 (+ 1269us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.156350 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:55.156730 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.156736 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.156840 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.157413 (+ 573us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:55.157427 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:55.159289 (+ 1862us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":119,"threads_started":1}
W20260430 20:02:55.159614 20308 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:55.153666 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:41265 (local address 127.10.46.62:41139)
0430 20:02:55.153845 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:55.153848 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:55.153953 (+ 105us) server_negotiation.cc:408] Connection header received
0430 20:02:55.154221 (+ 268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:55.154227 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:55.154286 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:55.154365 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:55.155208 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.156220 (+ 1012us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.156873 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.157015 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.159461 (+ 2446us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:41265: BlockingRecv error: recv got EOF from 127.10.46.3:41265 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:55.789031 20311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:55.780039 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:33179)
0430 20:02:55.780322 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:55.780337 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:55.780438 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:55.780999 (+ 561us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:55.781002 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:55.781220 (+ 218us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:55.781429 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.781435 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.782255 (+ 820us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.782258 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:55.782656 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.782662 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.782773 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.786616 (+ 3843us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:55.786634 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:55.788879 (+ 2245us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
W20260430 20:02:55.789256 20312 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:55.780129 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:33179 (local address 127.10.46.62:41139)
0430 20:02:55.780704 (+ 575us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:55.780708 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:55.780725 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:55.780763 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:55.780766 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:55.780812 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:55.780906 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:55.781585 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.782107 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.782823 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.783019 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.789108 (+ 6089us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:33179: BlockingRecv error: recv got EOF from 127.10.46.1:33179 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":193,"threads_started":1}
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:55 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:55.977556 20313 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:55.971201 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:47243)
0430 20:02:55.971696 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:55.971713 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:55.971907 (+ 194us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:55.972192 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:55.972196 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:55.972417 (+ 221us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:55.972675 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.972682 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.973445 (+ 763us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.973448 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:55.973825 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:55.973831 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.973961 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.974708 (+ 747us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:55.974729 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:55.977423 (+ 2694us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":101,"threads_started":1}
W20260430 20:02:55.977938 20312 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:55.971355 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:47243 (local address 127.10.46.62:41139)
0430 20:02:55.971673 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:55.971678 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:55.971835 (+ 157us) server_negotiation.cc:408] Connection header received
0430 20:02:55.972014 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:55.972018 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:55.972077 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:55.972175 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:55.972796 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.973303 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:55.973954 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:55.974160 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:55.977635 (+ 3475us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:47243: BlockingRecv error: recv got EOF from 127.10.46.2:47243 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:56.166065 20314 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:56.160591 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:48797)
0430 20:02:56.160872 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:56.160886 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:56.160985 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:56.161309 (+ 324us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:56.161315 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:56.161532 (+ 217us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:56.161736 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.161741 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.162508 (+ 767us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.162513 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:56.162906 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.162913 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.163043 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.163649 (+ 606us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:56.163664 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:56.165940 (+ 2276us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260430 20:02:56.166325 20312 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:56.160755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:48797 (local address 127.10.46.62:41139)
0430 20:02:56.160997 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:56.161028 (+ 31us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:56.161047 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:02:56.161118 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:56.161121 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:56.161170 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:56.161255 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:56.161873 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.162391 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.163026 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.163233 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.166211 (+ 2978us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:48797: BlockingRecv error: recv got EOF from 127.10.46.3:48797 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:56.796264 20315 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:56.789814 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:59351)
0430 20:02:56.790129 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:56.790145 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:56.790239 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:56.790639 (+ 400us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:56.790644 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:56.790933 (+ 289us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:56.791248 (+ 315us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.791256 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.792329 (+ 1073us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.792334 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:56.792962 (+ 628us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.792977 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.793151 (+ 174us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.793749 (+ 598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:56.793763 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:56.796138 (+ 2375us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
W20260430 20:02:56.796779 20316 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:56.789952 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:59351 (local address 127.10.46.62:41139)
0430 20:02:56.790202 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:56.790205 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:56.790219 (+ 14us) server_negotiation.cc:408] Connection header received
0430 20:02:56.790329 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:56.790331 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:56.790377 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:56.790460 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:56.791485 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.792133 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.793167 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.793348 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.796670 (+ 3322us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:59351: BlockingRecv error: recv got EOF from 127.10.46.1:59351 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:56 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:56.984324 20317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:56.978758 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:49827)
0430 20:02:56.979227 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:56.979247 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:56.979414 (+ 167us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:56.979740 (+ 326us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:56.979743 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:56.979961 (+ 218us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:56.980159 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.980165 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.980967 (+ 802us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.980970 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:56.981413 (+ 443us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:56.981420 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.981538 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.982140 (+ 602us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:56.982154 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:56.984141 (+ 1987us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":176,"threads_started":1}
W20260430 20:02:56.984429 20316 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:56.978863 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:49827 (local address 127.10.46.62:41139)
0430 20:02:56.979080 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:56.979086 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:56.979401 (+ 315us) server_negotiation.cc:408] Connection header received
0430 20:02:56.979514 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:56.979517 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:56.979572 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:56.979658 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:56.980316 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.980832 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:56.981611 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:56.981829 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:56.984308 (+ 2479us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:49827: BlockingRecv error: recv got EOF from 127.10.46.2:49827 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:57.172626 20318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:57.167076 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:39475)
0430 20:02:57.167538 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:57.167559 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:57.167721 (+ 162us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:57.168069 (+ 348us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:57.168072 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:57.168296 (+ 224us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:57.168487 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.168492 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.169242 (+ 750us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.169247 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:57.169633 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.169639 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.169771 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.170382 (+ 611us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:57.170396 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:57.172503 (+ 2107us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":164,"threads_started":1}
W20260430 20:02:57.173080 20316 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:57.167321 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:39475 (local address 127.10.46.62:41139)
0430 20:02:57.167538 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:57.167542 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:57.167654 (+ 112us) server_negotiation.cc:408] Connection header received
0430 20:02:57.167886 (+ 232us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:57.167895 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:57.167945 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:57.168038 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:57.168609 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.169125 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.169757 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.169948 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.172924 (+ 2976us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:39475: BlockingRecv error: recv got EOF from 127.10.46.3:39475 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:57.804199 20319 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:57.797278 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:37019)
0430 20:02:57.797654 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:57.797670 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:57.797777 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:57.798388 (+ 611us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:57.798391 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:57.798585 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:57.798824 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.798829 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.800028 (+ 1199us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.800033 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:57.800766 (+ 733us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.800778 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.800914 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.801569 (+ 655us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:57.801590 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:57.804061 (+ 2471us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":106,"threads_started":1}
W20260430 20:02:57.804461 20320 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:57.797425 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:37019 (local address 127.10.46.62:41139)
0430 20:02:57.797960 (+ 535us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:57.797965 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:57.797987 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:02:57.798046 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:57.798049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:57.798112 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:57.798473 (+ 361us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:57.798994 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.799712 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.800917 (+ 1205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.801175 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.804296 (+ 3121us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:37019: BlockingRecv error: recv got EOF from 127.10.46.1:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":139,"threads_started":1}
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:57 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:57.990691 20321 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:57.985225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:45763)
0430 20:02:57.985753 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:57.985771 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:57.985901 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:57.986195 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:57.986198 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:57.986424 (+ 226us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:57.986613 (+ 189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.986618 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.987390 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.987396 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:57.987856 (+ 460us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:57.987864 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.987992 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.988583 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:57.988597 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:57.990566 (+ 1969us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":127,"threads_started":1}
W20260430 20:02:57.990917 20320 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:57.985347 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:45763 (local address 127.10.46.62:41139)
0430 20:02:57.985507 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:57.985511 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:57.985857 (+ 346us) server_negotiation.cc:408] Connection header received
0430 20:02:57.986013 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:57.986017 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:57.986079 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:57.986171 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:57.986749 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.987265 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:57.987970 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:57.988176 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:57.990785 (+ 2609us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:45763: BlockingRecv error: recv got EOF from 127.10.46.2:45763 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:58.178591 20322 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:58.173457 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:46989)
0430 20:02:58.173734 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:58.173748 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:58.173858 (+ 110us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:58.174112 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:58.174115 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:58.174309 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:58.174512 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.174517 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.175292 (+ 775us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.175296 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:58.175698 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.175705 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.175906 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.176500 (+ 594us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:58.176514 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:58.178467 (+ 1953us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
W20260430 20:02:58.178783 20320 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:58.173588 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:46989 (local address 127.10.46.62:41139)
0430 20:02:58.173751 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:58.173755 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:58.173822 (+ 67us) server_negotiation.cc:408] Connection header received
0430 20:02:58.173939 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:58.173942 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:58.173991 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:58.174060 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:58.174627 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.175119 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.175907 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.176060 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.178635 (+ 2575us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:46989: BlockingRecv error: recv got EOF from 127.10.46.3:46989 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:58.811249 20323 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:58.805189 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:50177)
0430 20:02:58.805726 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:58.805752 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:58.805939 (+ 187us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:58.806341 (+ 402us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:58.806345 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:58.806560 (+ 215us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:58.806777 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.806783 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.807668 (+ 885us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.807675 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:58.808136 (+ 461us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.808144 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.808261 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.808973 (+ 712us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:58.808997 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:58.811037 (+ 2040us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":91,"threads_started":1}
W20260430 20:02:58.811287 20324 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:58.805539 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50177 (local address 127.10.46.62:41139)
0430 20:02:58.805937 (+ 398us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:58.805941 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:58.805961 (+ 20us) server_negotiation.cc:408] Connection header received
0430 20:02:58.806062 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:58.806065 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:58.806122 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:58.806230 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:58.806949 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.807522 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.808281 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.808506 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.811168 (+ 2662us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:50177: BlockingRecv error: recv got EOF from 127.10.46.1:50177 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:58 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:58.997049 20325 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:58.991532 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:34015)
0430 20:02:58.991868 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:58.991889 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:58.992033 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:58.992331 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:58.992334 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:58.992543 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:58.992739 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.992744 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.993514 (+ 770us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.993521 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:58.993910 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:58.993917 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.994054 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.994714 (+ 660us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:58.994727 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:58.996914 (+ 2187us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":94,"threads_started":1}
W20260430 20:02:58.997371 20324 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:58.991585 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:34015 (local address 127.10.46.62:41139)
0430 20:02:58.991829 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:58.991834 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:58.991967 (+ 133us) server_negotiation.cc:408] Connection header received
0430 20:02:58.992139 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:58.992143 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:58.992195 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:58.992310 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:58.992869 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.993396 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:58.994061 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:58.994277 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:58.997086 (+ 2809us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:34015: BlockingRecv error: recv got EOF from 127.10.46.2:34015 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:59.184753 20326 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:59.179534 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:37495)
0430 20:02:59.179838 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:59.179852 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:59.179959 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:59.180224 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:59.180226 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:59.180412 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:59.180608 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:59.180614 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.181511 (+ 897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:59.181514 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:59.181910 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:59.181917 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.182037 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:59.182660 (+ 623us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:59.182674 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:59.184628 (+ 1954us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
W20260430 20:02:59.184934 20324 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:59.179698 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:37495 (local address 127.10.46.62:41139)
0430 20:02:59.179873 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:59.179876 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:59.179907 (+ 31us) server_negotiation.cc:408] Connection header received
0430 20:02:59.180048 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:59.180051 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:59.180100 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:59.180184 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:59.180734 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.181247 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:59.182048 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.182194 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:59.184812 (+ 2618us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:37495: BlockingRecv error: recv got EOF from 127.10.46.3:37495 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:02:59.818740 20327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:59.812405 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:57241)
0430 20:02:59.812725 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:59.812762 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:59.812882 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:59.813209 (+ 327us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:59.813213 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:59.813493 (+ 280us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:59.813716 (+ 223us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:59.813722 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.814631 (+ 909us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:02:59.814641 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:02:59.815045 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:59.815052 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.815191 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:59.815977 (+ 786us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:02:59.815991 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:02:59.818614 (+ 2623us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
W20260430 20:02:59.818971 20328 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:59.812480 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:57241 (local address 127.10.46.62:41139)
0430 20:02:59.812876 (+ 396us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:59.812881 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:59.812898 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:02:59.813035 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:59.813039 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:59.813087 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:59.813188 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:02:59.813840 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.814417 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:02:59.815173 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:02:59.815397 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:02:59.818794 (+ 3397us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:57241: BlockingRecv error: recv got EOF from 127.10.46.1:57241 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":63,"threads_started":1}
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:02:59 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:00.004415 20329 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:02:59.998452 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:35803)
0430 20:02:59.998999 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0430 20:02:59.999018 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:02:59.999154 (+ 136us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:02:59.999466 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:02:59.999469 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:02:59.999735 (+ 266us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:02:59.999966 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:02:59.999972 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.000730 (+ 758us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.000737 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:00.001136 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:00.001143 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.001280 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.002109 (+ 829us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:00.002123 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:00.004278 (+ 2155us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":119,"threads_started":1}
W20260430 20:03:00.004628 20328 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:02:59.998858 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:35803 (local address 127.10.46.62:41139)
0430 20:02:59.999031 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0430 20:02:59.999036 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:02:59.999110 (+ 74us) server_negotiation.cc:408] Connection header received
0430 20:02:59.999258 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:02:59.999261 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:02:59.999335 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:02:59.999444 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:00.000096 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.000614 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.001257 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.001466 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.004508 (+ 3042us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:35803: BlockingRecv error: recv got EOF from 127.10.46.2:35803 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Apr 30 20:03:00 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:00 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:00.191188 20330 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:00.185877 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:45621)
0430 20:03:00.186360 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:00.186375 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:00.186506 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:00.186786 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:00.186789 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:00.187009 (+ 220us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:00.187201 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:00.187206 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.188015 (+ 809us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.188019 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:00.188403 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:00.188409 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.188544 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.189115 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:00.189135 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:00.191020 (+ 1885us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":108,"threads_started":1}
W20260430 20:03:00.191226 20328 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:00.185943 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:45621 (local address 127.10.46.62:41139)
0430 20:03:00.186099 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:00.186103 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:00.186448 (+ 345us) server_negotiation.cc:408] Connection header received
0430 20:03:00.186611 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:00.186615 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:00.186672 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:00.186759 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:00.187330 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.187888 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.188522 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.188712 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.191122 (+ 2410us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:45621: BlockingRecv error: recv got EOF from 127.10.46.3:45621 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 30 20:03:00 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:00 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:00.831831 20331 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:00.820012 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:53839)
0430 20:03:00.820471 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:00.820487 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:00.820577 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:00.820873 (+ 296us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:00.820876 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:00.821212 (+ 336us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:00.821426 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:00.821433 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.822231 (+ 798us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.822237 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:00.822630 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:00.822636 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.822771 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.829014 (+ 6243us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:00.829040 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:00.831602 (+ 2562us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":158,"threads_started":1}
W20260430 20:03:00.831960 20332 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:00.820049 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:53839 (local address 127.10.46.62:41139)
0430 20:03:00.820521 (+ 472us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:00.820526 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:00.820542 (+ 16us) server_negotiation.cc:408] Connection header received
0430 20:03:00.820695 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:00.820699 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:00.820755 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:00.820854 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:00.821570 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.822108 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:00.822759 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:00.822966 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:00.831801 (+ 8835us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:53839: BlockingRecv error: recv got EOF from 127.10.46.1:53839 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":75,"threads_started":1}
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:01.011723 20333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:01.005525 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:56707)
0430 20:03:01.005844 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:01.005858 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:01.005985 (+ 127us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:01.006326 (+ 341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:01.006332 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:01.006529 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:01.006727 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.006732 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.007844 (+ 1112us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.007848 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:01.008356 (+ 508us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.008366 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.008516 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.009221 (+ 705us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:01.009236 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:01.011595 (+ 2359us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
W20260430 20:03:01.012135 20332 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:01.005624 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:56707 (local address 127.10.46.62:41139)
0430 20:03:01.005838 (+ 214us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:01.005841 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:01.006015 (+ 174us) server_negotiation.cc:408] Connection header received
0430 20:03:01.006111 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:01.006114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:01.006168 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:01.006250 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:01.006900 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.007642 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.008669 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.008966 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.012016 (+ 3050us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:56707: BlockingRecv error: recv got EOF from 127.10.46.2:56707 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.3@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:01.199501 20334 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:01.192250 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.3:50853)
0430 20:03:01.192636 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:01.192650 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:01.192769 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:01.193129 (+ 360us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:01.193132 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:01.193338 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:01.193547 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.193552 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.194557 (+ 1005us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.194561 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:01.195151 (+ 590us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.195189 (+ 38us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.195319 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.196013 (+ 694us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:01.196028 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:01.199308 (+ 3280us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":175,"threads_started":1}
W20260430 20:03:01.199546 20332 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:01.192438 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:50853 (local address 127.10.46.62:41139)
0430 20:03:01.192587 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:01.192590 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:01.192760 (+ 170us) server_negotiation.cc:408] Connection header received
0430 20:03:01.192911 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:01.192916 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:01.192978 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:01.193071 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:01.193799 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.194395 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.195347 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.195498 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.199445 (+ 3947us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.3:50853: BlockingRecv error: recv got EOF from 127.10.46.3:50853 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:01 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.1@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:01.839218 20335 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:01.832991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.1:49519)
0430 20:03:01.833312 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:01.833330 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:01.833490 (+ 160us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:01.833924 (+ 434us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:01.833928 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:01.834301 (+ 373us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:01.834520 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.834527 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.835429 (+ 902us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.835433 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:01.836030 (+ 597us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:01.836039 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.836150 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.836759 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:01.836774 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:01.839041 (+ 2267us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":67,"threads_started":1}
W20260430 20:03:01.839570 20336 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:01.833140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:49519 (local address 127.10.46.62:41139)
0430 20:03:01.833506 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:01.833510 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:01.833527 (+ 17us) server_negotiation.cc:408] Connection header received
0430 20:03:01.833572 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:01.833575 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:01.833654 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:01.833757 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:01.834687 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.835282 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:01.836280 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:01.836507 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:01.839403 (+ 2896us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.1:49519: BlockingRecv error: recv got EOF from 127.10.46.1:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
Apr 30 20:03:02 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
Apr 30 20:03:02 dist-test-slave-0bdw krb5kdc[19268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.46.2@KRBTEST.COM for oryx/127.10.46.62@KRBTEST.COM, Server not found in Kerberos database
I20260430 20:03:02.019272 20337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.013001 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:41139 (local address 127.10.46.2:58097)
0430 20:03:02.013323 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:02.013338 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:02.013443 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:02.013743 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:02.013747 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:02.013964 (+ 217us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:02.014186 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.014192 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.015000 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.015004 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:02.015579 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.015588 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.015700 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.016432 (+ 732us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:02.016450 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:02.019112 (+ 2662us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.46.62:41139: Server oryx/127.10.46.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
W20260430 20:03:02.019636 20336 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:02.013074 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:58097 (local address 127.10.46.62:41139)
0430 20:03:02.013304 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:02.013308 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:02.013447 (+ 139us) server_negotiation.cc:408] Connection header received
0430 20:03:02.013530 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:02.013533 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:02.013584 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:02.013669 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:02.014321 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.014821 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.015761 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.016050 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.019509 (+ 3459us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.46.2:58097: BlockingRecv error: recv got EOF from 127.10.46.2:58097 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260430 20:03:02.102176 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19860
I20260430 20:03:02.108984 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20012
I20260430 20:03:02.115377 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20146
I20260430 20:03:02.121944 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 19790
2026-04-30T20:03:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19199 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-04-30T20:03:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:03:02Z Disabled control of system clock
I20260430 20:03:02.157347 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34839
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:34839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260430 20:03:02.278806 20341 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:02.279063 20341 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:02.279106 20341 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:02.282949 20341 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:03:02.283048 20341 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:02.283066 20341 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:02.283087 20341 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:03:02.283104 20341 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:03:02.287729 20341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:34839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:34839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:02.289108 20341 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:02.290436 20341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:02.297690 20347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.297690 20346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.297745 20341 server_base.cc:1061] running on GCE node
W20260430 20:03:02.297690 20349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.299250 20341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:02.301021 20341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:02.302289 20341 hybrid_clock.cc:648] HybridClock initialized: now 1777579382302259 us; error 47 us; skew 500 ppm
I20260430 20:03:02.304790 20341 webserver.cc:492] Webserver started at http://127.10.46.62:44231/ using document root <none> and password file <none>
I20260430 20:03:02.305485 20341 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:02.305576 20341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:02.305800 20341 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:02.307633 20341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "bca586b88c6540039ad293ef835c5500"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "651b2979d39094667cfae5607df8c646"
server_key_iv: "bc38a7d1c95fe4bbe09d6285e607f3c8"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.308179 20341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "bca586b88c6540039ad293ef835c5500"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "651b2979d39094667cfae5607df8c646"
server_key_iv: "bc38a7d1c95fe4bbe09d6285e607f3c8"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.311602 20341 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:03:02.314393 20355 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.315955 20341 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 20:03:02.316152 20341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "bca586b88c6540039ad293ef835c5500"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "651b2979d39094667cfae5607df8c646"
server_key_iv: "bc38a7d1c95fe4bbe09d6285e607f3c8"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.316380 20341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:02.330858 20341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:02.331586 20341 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:02.331877 20341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:02.344077 20341 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:34839
I20260430 20:03:02.344111 20407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:34839 every 8 connection(s)
I20260430 20:03:02.345850 20341 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:03:02.346410 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20341
I20260430 20:03:02.346547 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:03:02.346814 10424 external_mini_cluster.cc:1468] Setting key 4f310353f9babe4c56d0cf4a57d2ec6c
I20260430 20:03:02.349244 20408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:02.355700 20408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: Bootstrap starting.
I20260430 20:03:02.355919 20411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.348126 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.10.46.62:34839)
0430 20:03:02.348711 (+ 585us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:02.348720 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:02.348753 (+ 33us) server_negotiation.cc:408] Connection header received
0430 20:03:02.349477 (+ 724us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:02.349495 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:02.349826 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:02.350114 (+ 288us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:02.350802 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.352140 (+ 1338us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.353167 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.353482 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.354018 (+ 536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:02.354097 (+ 79us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:02.354111 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:02.354542 (+ 431us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:02.354580 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:02.354595 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:02.354682 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:02.354972 (+ 290us) server_negotiation.cc:300] Negotiation successful
0430 20:03:02.355113 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
I20260430 20:03:02.360585 20408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:02.361989 20408 log.cc:826] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: Log is configured to *not* fsync() on all Append() calls
I20260430 20:03:02.365100 20408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: No bootstrap required, opened a new log
I20260430 20:03:02.369305 20408 raft_consensus.cc:359] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } }
I20260430 20:03:02.369587 20408 raft_consensus.cc:385] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:02.369635 20408 raft_consensus.cc:740] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca586b88c6540039ad293ef835c5500, State: Initialized, Role: FOLLOWER
I20260430 20:03:02.370191 20408 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } }
I20260430 20:03:02.370359 20408 raft_consensus.cc:399] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:03:02.370406 20408 raft_consensus.cc:493] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:03:02.370476 20408 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:02.371870 20408 raft_consensus.cc:515] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } }
I20260430 20:03:02.372277 20408 leader_election.cc:304] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca586b88c6540039ad293ef835c5500; no voters:
I20260430 20:03:02.372584 20408 leader_election.cc:290] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:03:02.372776 20413 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:03:02.373086 20413 raft_consensus.cc:697] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [term 1 LEADER]: Becoming Leader. State: Replica: bca586b88c6540039ad293ef835c5500, State: Running, Role: LEADER
I20260430 20:03:02.373616 20413 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } }
I20260430 20:03:02.373898 20408 sys_catalog.cc:565] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:03:02.376161 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca586b88c6540039ad293ef835c5500. Latest consensus state: current_term: 1 leader_uuid: "bca586b88c6540039ad293ef835c5500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } } }
I20260430 20:03:02.376369 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [sys.catalog]: This master's current role is: LEADER
I20260430 20:03:02.377593 20414 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca586b88c6540039ad293ef835c5500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca586b88c6540039ad293ef835c5500" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 34839 } } }
I20260430 20:03:02.377728 20414 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500 [sys.catalog]: This master's current role is: LEADER
W20260430 20:03:02.381433 20428 catalog_manager.cc:1568] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 20:03:02.381518 20428 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 20:03:02.381613 20422 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:03:02.382886 20422 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:03:02.389426 20422 catalog_manager.cc:1357] Generated new cluster ID: b90dfdced7c24892b231d0d938153e91
I20260430 20:03:02.389534 20422 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:03:02.400045 20422 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:03:02.401314 20422 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:03:02.414304 20422 catalog_manager.cc:6044] T 00000000000000000000000000000000 P bca586b88c6540039ad293ef835c5500: Generated new TSK 0
I20260430 20:03:02.415035 20422 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:03:02.421413 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--builtin_ntp_servers=127.10.46.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260430 20:03:02.555910 20432 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:02.556152 20432 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:02.556200 20432 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:02.560271 20432 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:02.560352 20432 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:02.560429 20432 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:03:02.565109 20432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:02.566349 20432 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:02.567571 20432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:02.575599 20440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.575598 20437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.575906 20438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.575999 20432 server_base.cc:1061] running on GCE node
I20260430 20:03:02.576530 20432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:02.577240 20432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:02.578514 20432 hybrid_clock.cc:648] HybridClock initialized: now 1777579382578486 us; error 294 us; skew 500 ppm
I20260430 20:03:02.580880 20432 webserver.cc:492] Webserver started at http://127.10.46.1:38981/ using document root <none> and password file <none>
I20260430 20:03:02.581670 20432 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:02.581766 20432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:02.582129 20432 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:02.584213 20432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "f503d95c251d4fcab060c0af0b81e7e3"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "ee52a7954f4190b704aba0674a91c267"
server_key_iv: "840f18e7021d7e49812f50c953b02aeb"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.584743 20432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "f503d95c251d4fcab060c0af0b81e7e3"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "ee52a7954f4190b704aba0674a91c267"
server_key_iv: "840f18e7021d7e49812f50c953b02aeb"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.588482 20432 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 20:03:02.591338 20446 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.592730 20432 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260430 20:03:02.592912 20432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "f503d95c251d4fcab060c0af0b81e7e3"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "ee52a7954f4190b704aba0674a91c267"
server_key_iv: "840f18e7021d7e49812f50c953b02aeb"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.593079 20432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:02.613862 20432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:02.614724 20432 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:02.615124 20432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:02.615890 20432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:02.617043 20432 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:02.617120 20432 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.617189 20432 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:02.617234 20432 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.627702 20432 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:35557
I20260430 20:03:02.627738 20559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:35557 every 8 connection(s)
I20260430 20:03:02.628744 20432 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:03:02.630335 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20432
I20260430 20:03:02.630478 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:03:02.630770 10424 external_mini_cluster.cc:1468] Setting key c4788dbf656bba9d2e818a4d60bbe84d
I20260430 20:03:02.633030 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--builtin_ntp_servers=127.10.46.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260430 20:03:02.638284 20411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.630628 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:60439 (local address 127.10.46.62:34839)
0430 20:03:02.630767 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:02.630771 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:02.631922 (+ 1151us) server_negotiation.cc:408] Connection header received
0430 20:03:02.632971 (+ 1049us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:02.632975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:02.633036 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:02.633136 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:02.634085 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.634749 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.635838 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.636145 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.637461 (+ 1316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:02.637491 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:02.637498 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:02.637520 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:02.637549 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:02.637562 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:02.637673 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:02.637892 (+ 219us) server_negotiation.cc:300] Negotiation successful
0430 20:03:02.638038 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260430 20:03:02.638909 20562 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.630965 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34839 (local address 127.10.46.1:60439)
0430 20:03:02.631757 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:02.631824 (+ 67us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:02.632721 (+ 897us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:02.633372 (+ 651us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:02.633385 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:02.633453 (+ 68us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:02.633929 (+ 476us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.633941 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.634903 (+ 962us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.634908 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:02.635581 (+ 673us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.635671 (+ 90us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.636080 (+ 409us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.637201 (+ 1121us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:02.637203 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:02.637232 (+ 29us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:02.637292 (+ 60us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:02.637715 (+ 423us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:02.637723 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:02.637727 (+ 4us) client_negotiation.cc:770] Sending connection context
0430 20:03:02.637871 (+ 144us) client_negotiation.cc:241] Negotiation successful
0430 20:03:02.638039 (+ 168us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"spinlock_wait_cycles":114048,"thread_start_us":123,"threads_started":1}
I20260430 20:03:02.640297 20560 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34839
I20260430 20:03:02.640625 20560 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:02.641276 20560 heartbeater.cc:507] Master 127.10.46.62:34839 requested a full tablet report, sending...
I20260430 20:03:02.643016 20372 ts_manager.cc:194] Registered new tserver with Master: f503d95c251d4fcab060c0af0b81e7e3 (127.10.46.1:35557)
I20260430 20:03:02.644843 20372 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.46.1:60439
W20260430 20:03:02.747151 20563 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:02.747437 20563 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:02.747491 20563 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:02.751169 20563 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:02.751245 20563 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:02.751318 20563 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:03:02.755471 20563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:02.756687 20563 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:02.757789 20563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:02.765715 20571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.765779 20569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.765786 20568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.766651 20563 server_base.cc:1061] running on GCE node
I20260430 20:03:02.767295 20563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:02.768463 20563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:02.769698 20563 hybrid_clock.cc:648] HybridClock initialized: now 1777579382769648 us; error 68 us; skew 500 ppm
I20260430 20:03:02.772035 20563 webserver.cc:492] Webserver started at http://127.10.46.2:44665/ using document root <none> and password file <none>
I20260430 20:03:02.772675 20563 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:02.772761 20563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:02.772985 20563 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:02.774848 20563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "b93e379bbb974e438837cdcfdcd59281"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "e0a03b4234aabca7be15509096b05915"
server_key_iv: "bb0a82181424fcd645849443dd61404b"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.775364 20563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "b93e379bbb974e438837cdcfdcd59281"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "e0a03b4234aabca7be15509096b05915"
server_key_iv: "bb0a82181424fcd645849443dd61404b"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.779325 20563 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260430 20:03:02.781936 20577 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.783010 20563 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:03:02.783164 20563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "b93e379bbb974e438837cdcfdcd59281"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "e0a03b4234aabca7be15509096b05915"
server_key_iv: "bb0a82181424fcd645849443dd61404b"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.783288 20563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:02.797142 20563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:02.797897 20563 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:02.798128 20563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:02.798763 20563 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:02.799872 20563 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:02.799960 20563 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.800038 20563 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:02.800091 20563 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.810396 20563 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:43925
I20260430 20:03:02.810549 20690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:43925 every 8 connection(s)
I20260430 20:03:02.811460 20563 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:03:02.820173 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20563
I20260430 20:03:02.820235 20411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.813524 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:39433 (local address 127.10.46.62:34839)
0430 20:03:02.813727 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:02.813732 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:02.814549 (+ 817us) server_negotiation.cc:408] Connection header received
0430 20:03:02.815442 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:02.815446 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:02.815512 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:02.815609 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:02.816549 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.817064 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.817861 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.818053 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.819583 (+ 1530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:02.819612 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:02.819615 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:02.819629 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:02.819650 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:02.819658 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:02.819743 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:02.819979 (+ 236us) server_negotiation.cc:300] Negotiation successful
0430 20:03:02.820051 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:03:02.820444 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:03:02.820725 10424 external_mini_cluster.cc:1468] Setting key ca8a11681e80968d943f7ababc9a733f
I20260430 20:03:02.821025 20693 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:02.813770 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34839 (local address 127.10.46.2:39433)
0430 20:03:02.814377 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:02.814411 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:02.815242 (+ 831us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:02.815818 (+ 576us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:02.815831 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:02.815899 (+ 68us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:02.816394 (+ 495us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.816405 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.817193 (+ 788us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:02.817197 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:02.817712 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:02.817723 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:02.818033 (+ 310us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:02.819334 (+ 1301us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:02.819335 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:02.819363 (+ 28us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:02.819415 (+ 52us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:02.819749 (+ 334us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:02.819758 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:02.819819 (+ 61us) client_negotiation.cc:770] Sending connection context
0430 20:03:02.819988 (+ 169us) client_negotiation.cc:241] Negotiation successful
0430 20:03:02.820151 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20260430 20:03:02.822732 20691 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34839
I20260430 20:03:02.823009 20691 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:02.823480 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--builtin_ntp_servers=127.10.46.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260430 20:03:02.823706 20691 heartbeater.cc:507] Master 127.10.46.62:34839 requested a full tablet report, sending...
I20260430 20:03:02.824990 20372 ts_manager.cc:194] Registered new tserver with Master: b93e379bbb974e438837cdcfdcd59281 (127.10.46.2:43925)
I20260430 20:03:02.825608 20372 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.46.2:39433
W20260430 20:03:02.930531 20694 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:02.930794 20694 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:02.930894 20694 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:02.934413 20694 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:02.934510 20694 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:02.934612 20694 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:03:02.939256 20694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:34839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:02.940454 20694 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:02.941594 20694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:02.948418 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.948581 20694 server_base.cc:1061] running on GCE node
W20260430 20:03:02.948459 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:02.948484 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:02.949150 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:02.949795 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:02.951013 20694 hybrid_clock.cc:648] HybridClock initialized: now 1777579382950971 us; error 55 us; skew 500 ppm
I20260430 20:03:02.953049 20694 webserver.cc:492] Webserver started at http://127.10.46.3:41089/ using document root <none> and password file <none>
I20260430 20:03:02.953645 20694 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:02.953728 20694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:02.953985 20694 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:02.955755 20694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "aa24afc82c624f2eb8ef5bb1c3ddcd10"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "a27c14a7e9464c79be74ec0ddc1681ff"
server_key_iv: "fe70be8a9907e60aabb2ced42e55c77c"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.956369 20694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "aa24afc82c624f2eb8ef5bb1c3ddcd10"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "a27c14a7e9464c79be74ec0ddc1681ff"
server_key_iv: "fe70be8a9907e60aabb2ced42e55c77c"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.959741 20694 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260430 20:03:02.962316 20708 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.963377 20694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:03:02.963517 20694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "aa24afc82c624f2eb8ef5bb1c3ddcd10"
format_stamp: "Formatted at 2026-04-30 20:03:02 on dist-test-slave-0bdw"
server_key: "a27c14a7e9464c79be74ec0ddc1681ff"
server_key_iv: "fe70be8a9907e60aabb2ced42e55c77c"
server_key_version: "encryptionkey@0"
I20260430 20:03:02.963635 20694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:02.990890 20694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:02.991640 20694 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:02.991920 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:02.992563 20694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:02.993670 20694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:02.993746 20694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:02.993821 20694 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:02.993871 20694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.003764 20694 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:38267
I20260430 20:03:03.003818 20821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:38267 every 8 connection(s)
I20260430 20:03:03.004768 20694 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:03:03.010649 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20694
I20260430 20:03:03.010762 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:03:03.011039 10424 external_mini_cluster.cc:1468] Setting key 88563e8dc36c6653945ec627f63cabd5
I20260430 20:03:03.013303 20411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.006615 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:55963 (local address 127.10.46.62:34839)
0430 20:03:03.006876 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.006880 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.007733 (+ 853us) server_negotiation.cc:408] Connection header received
0430 20:03:03.008629 (+ 896us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.008632 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.008685 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.008780 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.009648 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:03.010158 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:03.010959 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:03.011143 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:03.012724 (+ 1581us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:03.012740 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:03.012743 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:03.012756 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:03.012779 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:03.012788 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:03.012867 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:03.013090 (+ 223us) server_negotiation.cc:300] Negotiation successful
0430 20:03:03.013150 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260430 20:03:03.013962 20824 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.006859 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:34839 (local address 127.10.46.3:55963)
0430 20:03:03.007573 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:03.007619 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:03.008424 (+ 805us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:03.008955 (+ 531us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:03.008963 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:03.009021 (+ 58us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:03.009505 (+ 484us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:03.009515 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:03.010309 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:03.010312 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:03.010821 (+ 509us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:03.010835 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:03.011091 (+ 256us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:03.012450 (+ 1359us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:03.012453 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:03.012486 (+ 33us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:03.012550 (+ 64us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:03.012889 (+ 339us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:03.012896 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:03.012902 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 20:03:03.013087 (+ 185us) client_negotiation.cc:241] Negotiation successful
0430 20:03:03.013245 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":108,"threads_started":1}
I20260430 20:03:03.014894 20822 heartbeater.cc:344] Connected to a master server at 127.10.46.62:34839
I20260430 20:03:03.015132 20822 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:03.015647 20822 heartbeater.cc:507] Master 127.10.46.62:34839 requested a full tablet report, sending...
I20260430 20:03:03.016676 20372 ts_manager.cc:194] Registered new tserver with Master: aa24afc82c624f2eb8ef5bb1c3ddcd10 (127.10.46.3:38267)
I20260430 20:03:03.017230 20372 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.46.3:55963
I20260430 20:03:03.025488 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 20:03:03.028537 20411 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:03.027546 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.10.46.62:34839)
0430 20:03:03.027715 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.027720 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.027865 (+ 145us) server_negotiation.cc:408] Connection header received
0430 20:03:03.028027 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.028031 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.028083 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.028172 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.028429 (+ 257us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50652: BlockingRecv error: recv got EOF from 127.0.0.1:50652 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 20:03:03.028877 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20432
I20260430 20:03:03.035135 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20563
I20260430 20:03:03.041281 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20694
I20260430 20:03:03.047392 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20341
2026-04-30T20:03:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (921 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-04-30T20:03:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:03:03Z Disabled control of system clock
I20260430 20:03:03.077039 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:32993
--webserver_interface=127.10.46.62
--webserver_port=0
--builtin_ntp_servers=127.10.46.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260430 20:03:03.188621 20833 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:03.188884 20833 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:03.188946 20833 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:03.192407 20833 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:03:03.192480 20833 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:03.192497 20833 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:03.192517 20833 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:03:03.192535 20833 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:03:03.196930 20833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:32993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:03.198077 20833 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:03.199324 20833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:03.206295 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.206281 20841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.206377 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:03.207396 20833 server_base.cc:1061] running on GCE node
I20260430 20:03:03.208536 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:03.209707 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:03.210995 20833 hybrid_clock.cc:648] HybridClock initialized: now 1777579383210961 us; error 50 us; skew 500 ppm
I20260430 20:03:03.213860 20833 webserver.cc:492] Webserver started at http://127.10.46.62:35313/ using document root <none> and password file <none>
I20260430 20:03:03.214553 20833 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:03.214658 20833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:03.214890 20833 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:03.216912 20833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "4405f558b11f484886ba1f855d165be9"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "be4388e7f8b5a41181e5218d577d2796"
server_key_iv: "b1b0508ff006103dabdb3cede91b7757"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.217465 20833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "4405f558b11f484886ba1f855d165be9"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "be4388e7f8b5a41181e5218d577d2796"
server_key_iv: "b1b0508ff006103dabdb3cede91b7757"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.221167 20833 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:03:03.223984 20847 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.225126 20833 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260430 20:03:03.225324 20833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "4405f558b11f484886ba1f855d165be9"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "be4388e7f8b5a41181e5218d577d2796"
server_key_iv: "b1b0508ff006103dabdb3cede91b7757"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.225448 20833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:03.230777 20833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:03.231419 20833 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:03.231607 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:03.240360 20833 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:32993
I20260430 20:03:03.240410 20899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:32993 every 8 connection(s)
I20260430 20:03:03.241418 20833 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:03:03.243106 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20833
I20260430 20:03:03.243270 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:03:03.243532 10424 external_mini_cluster.cc:1468] Setting key 9469a2cdd29f8e3babcf0ba77d570dbc
I20260430 20:03:03.244526 20900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:03.248688 20903 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.244844 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.10.46.62:32993)
0430 20:03:03.245463 (+ 619us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.245471 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.245504 (+ 33us) server_negotiation.cc:408] Connection header received
0430 20:03:03.246141 (+ 637us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.246166 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.246454 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.246782 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.247622 (+ 840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:03.247627 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:03.247636 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:03.247647 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:03.247680 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:03.247689 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:03.247882 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:03.247976 (+ 94us) server_negotiation.cc:300] Negotiation successful
0430 20:03:03.248109 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":128,"threads_started":1}
I20260430 20:03:03.250509 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9: Bootstrap starting.
I20260430 20:03:03.252713 20900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:03.253399 20900 log.cc:826] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9: Log is configured to *not* fsync() on all Append() calls
I20260430 20:03:03.256346 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9: No bootstrap required, opened a new log
I20260430 20:03:03.258916 20900 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } }
I20260430 20:03:03.259135 20900 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:03.259187 20900 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4405f558b11f484886ba1f855d165be9, State: Initialized, Role: FOLLOWER
I20260430 20:03:03.259599 20900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } }
I20260430 20:03:03.259809 20900 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:03:03.259881 20900 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:03:03.259976 20900 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:03.261004 20900 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } }
I20260430 20:03:03.261348 20900 leader_election.cc:304] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4405f558b11f484886ba1f855d165be9; no voters:
I20260430 20:03:03.261619 20900 leader_election.cc:290] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:03:03.261762 20905 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:03:03.262015 20905 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [term 1 LEADER]: Becoming Leader. State: Replica: 4405f558b11f484886ba1f855d165be9, State: Running, Role: LEADER
I20260430 20:03:03.262339 20905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } }
I20260430 20:03:03.262782 20900 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:03:03.264175 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4405f558b11f484886ba1f855d165be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } } }
I20260430 20:03:03.264336 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [sys.catalog]: This master's current role is: LEADER
I20260430 20:03:03.264866 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4405f558b11f484886ba1f855d165be9. Latest consensus state: current_term: 1 leader_uuid: "4405f558b11f484886ba1f855d165be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4405f558b11f484886ba1f855d165be9" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 32993 } } }
I20260430 20:03:03.264993 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9 [sys.catalog]: This master's current role is: LEADER
I20260430 20:03:03.265169 20914 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:03:03.268306 20914 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:03:03.275326 20914 catalog_manager.cc:1357] Generated new cluster ID: ed90b7dc712e4708b97f3dfe54ddc1bd
I20260430 20:03:03.275435 20914 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:03:03.289315 20914 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:03:03.289830 20914 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:03:03.298905 20914 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4405f558b11f484886ba1f855d165be9: Generated new TSK 0
I20260430 20:03:03.299590 20914 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 20:03:03.307178 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--builtin_ntp_servers=127.10.46.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260430 20:03:03.414518 20924 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:03.414758 20924 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:03.414850 20924 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:03.418509 20924 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:03.418583 20924 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:03.418694 20924 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:03:03.422919 20924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:03.424158 20924 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:03.425356 20924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:03.432579 20929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.432827 20932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.433043 20930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:03.433058 20924 server_base.cc:1061] running on GCE node
I20260430 20:03:03.433542 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:03.434221 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:03.435413 20924 hybrid_clock.cc:648] HybridClock initialized: now 1777579383435389 us; error 40 us; skew 500 ppm
I20260430 20:03:03.437877 20924 webserver.cc:492] Webserver started at http://127.10.46.1:39371/ using document root <none> and password file <none>
I20260430 20:03:03.438584 20924 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:03.438660 20924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:03.438872 20924 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:03.441087 20924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "3eb266e7ccfb41a6b82a3d42021cdcde"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "ad9acb3ca1a6085d039e96956bcf29c9"
server_key_iv: "63183972bce4a148befe94f20c44f227"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.441591 20924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "3eb266e7ccfb41a6b82a3d42021cdcde"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "ad9acb3ca1a6085d039e96956bcf29c9"
server_key_iv: "63183972bce4a148befe94f20c44f227"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.445173 20924 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 20:03:03.448249 20938 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.449604 20924 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260430 20:03:03.449767 20924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "3eb266e7ccfb41a6b82a3d42021cdcde"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "ad9acb3ca1a6085d039e96956bcf29c9"
server_key_iv: "63183972bce4a148befe94f20c44f227"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.449888 20924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:03.454892 20924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:03.455660 20924 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:03.455925 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:03.456552 20924 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:03.457648 20924 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:03.457732 20924 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.457810 20924 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:03.457859 20924 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.468716 20924 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:44391
I20260430 20:03:03.468772 21051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:44391 every 8 connection(s)
I20260430 20:03:03.470454 20924 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:03:03.473477 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 20924
I20260430 20:03:03.473646 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:03:03.473912 10424 external_mini_cluster.cc:1468] Setting key 87b0e1168b8c227729b4bcbf41e503e3
I20260430 20:03:03.476282 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--builtin_ntp_servers=127.10.46.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260430 20:03:03.476980 20903 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.473028 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:56665 (local address 127.10.46.62:32993)
0430 20:03:03.473171 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.473175 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.474133 (+ 958us) server_negotiation.cc:408] Connection header received
0430 20:03:03.475077 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.475081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.475116 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.475223 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.476507 (+ 1284us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:03.476511 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:03.476514 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:03.476530 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:03.476551 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:03.476557 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:03.476654 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:03.476806 (+ 152us) server_negotiation.cc:300] Negotiation successful
0430 20:03:03.476844 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 20:03:03.477757 21054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.473265 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:32993 (local address 127.10.46.1:56665)
0430 20:03:03.473970 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:03.474016 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:03.474877 (+ 861us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:03.475517 (+ 640us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:03.475526 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:03.475592 (+ 66us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:03.476259 (+ 667us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:03.476261 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:03.476286 (+ 25us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:03.476336 (+ 50us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:03.476642 (+ 306us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:03.476648 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:03.476654 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 20:03:03.476853 (+ 199us) client_negotiation.cc:241] Negotiation successful
0430 20:03:03.477005 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260430 20:03:03.479417 21052 heartbeater.cc:344] Connected to a master server at 127.10.46.62:32993
I20260430 20:03:03.479703 21052 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:03.480243 21052 heartbeater.cc:507] Master 127.10.46.62:32993 requested a full tablet report, sending...
I20260430 20:03:03.481782 20864 ts_manager.cc:194] Registered new tserver with Master: 3eb266e7ccfb41a6b82a3d42021cdcde (127.10.46.1:44391)
W20260430 20:03:03.588680 21055 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:03.588973 21055 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:03.589061 21055 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:03.592842 21055 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:03.592983 21055 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:03.593115 21055 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:03:03.599956 21055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:03.601702 21055 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:03.603566 21055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:03.610530 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.610530 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.610739 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:03.610739 21055 server_base.cc:1061] running on GCE node
I20260430 20:03:03.611339 21055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:03.611991 21055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:03.613190 21055 hybrid_clock.cc:648] HybridClock initialized: now 1777579383613177 us; error 38 us; skew 500 ppm
I20260430 20:03:03.615363 21055 webserver.cc:492] Webserver started at http://127.10.46.2:46171/ using document root <none> and password file <none>
I20260430 20:03:03.616041 21055 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:03.616137 21055 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:03.616353 21055 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:03.618094 21055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "f50d8710a3f6433db37132aa2a3cfbdb"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "6ef6d1be84bf21ef0be7c06f7c66d0bf"
server_key_iv: "34432d2668e1b24fcafae3277407ac2f"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.618611 21055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "f50d8710a3f6433db37132aa2a3cfbdb"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "6ef6d1be84bf21ef0be7c06f7c66d0bf"
server_key_iv: "34432d2668e1b24fcafae3277407ac2f"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.622309 21055 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 20:03:03.625061 21069 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.626130 21055 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:03:03.626291 21055 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "f50d8710a3f6433db37132aa2a3cfbdb"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "6ef6d1be84bf21ef0be7c06f7c66d0bf"
server_key_iv: "34432d2668e1b24fcafae3277407ac2f"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.626410 21055 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:03.631557 21055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:03.632220 21055 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:03.632400 21055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:03.632988 21055 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:03.633977 21055 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:03.634045 21055 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.634122 21055 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:03.634159 21055 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.644351 21055 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:45673
I20260430 20:03:03.644383 21182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:45673 every 8 connection(s)
I20260430 20:03:03.645438 21055 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:03:03.650889 20903 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.647393 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:34991 (local address 127.10.46.62:32993)
0430 20:03:03.647553 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.647558 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.648463 (+ 905us) server_negotiation.cc:408] Connection header received
0430 20:03:03.649277 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.649280 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.649322 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.649406 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.650399 (+ 993us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:03.650403 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:03.650405 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:03.650416 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:03.650437 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:03.650445 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:03.650533 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:03.650739 (+ 206us) server_negotiation.cc:300] Negotiation successful
0430 20:03:03.650774 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 20:03:03.651530 21185 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.647682 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:32993 (local address 127.10.46.2:34991)
0430 20:03:03.648325 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:03.648358 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:03.649077 (+ 719us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:03.649551 (+ 474us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:03.649558 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:03.649620 (+ 62us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:03.650145 (+ 525us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:03.650147 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:03.650176 (+ 29us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:03.650226 (+ 50us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:03.650555 (+ 329us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:03.650562 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:03.650567 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 20:03:03.650734 (+ 167us) client_negotiation.cc:241] Negotiation successful
0430 20:03:03.650826 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260430 20:03:03.652987 21183 heartbeater.cc:344] Connected to a master server at 127.10.46.62:32993
I20260430 20:03:03.653234 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21055
I20260430 20:03:03.653251 21183 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:03.653371 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:03:03.653633 10424 external_mini_cluster.cc:1468] Setting key 44dcfb94ae950bc521cdea45564cfa95
I20260430 20:03:03.653730 21183 heartbeater.cc:507] Master 127.10.46.62:32993 requested a full tablet report, sending...
I20260430 20:03:03.654946 20860 ts_manager.cc:194] Registered new tserver with Master: f50d8710a3f6433db37132aa2a3cfbdb (127.10.46.2:45673)
I20260430 20:03:03.655948 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--builtin_ntp_servers=127.10.46.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260430 20:03:03.767289 21186 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:03.767527 21186 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:03.767620 21186 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:03.771284 21186 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:03.771370 21186 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:03.771494 21186 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:03:03.775899 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:03.777101 21186 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:03.778254 21186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:03.785341 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.785291 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:03.785291 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:03.785806 21186 server_base.cc:1061] running on GCE node
I20260430 20:03:03.786226 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:03.786793 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:03.787959 21186 hybrid_clock.cc:648] HybridClock initialized: now 1777579383787947 us; error 25 us; skew 500 ppm
I20260430 20:03:03.789803 21186 webserver.cc:492] Webserver started at http://127.10.46.3:32945/ using document root <none> and password file <none>
I20260430 20:03:03.790412 21186 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:03.790470 21186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:03.790671 21186 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:03.792475 21186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "8b5297ea0d734177a12b39f648c1ecbf"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "a050795bd529c11c0445dcdc84ddeecf"
server_key_iv: "0e9c49db7687c24afd6a89bc78cfbc23"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.792979 21186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "8b5297ea0d734177a12b39f648c1ecbf"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "a050795bd529c11c0445dcdc84ddeecf"
server_key_iv: "0e9c49db7687c24afd6a89bc78cfbc23"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.796396 21186 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 20:03:03.798774 21200 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.800033 21186 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260430 20:03:03.800174 21186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "8b5297ea0d734177a12b39f648c1ecbf"
format_stamp: "Formatted at 2026-04-30 20:03:03 on dist-test-slave-0bdw"
server_key: "a050795bd529c11c0445dcdc84ddeecf"
server_key_iv: "0e9c49db7687c24afd6a89bc78cfbc23"
server_key_version: "encryptionkey@0"
I20260430 20:03:03.800289 21186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:03.805357 21186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:03.805882 21186 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:03.806023 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:03.806505 21186 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:03.807442 21186 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:03.807498 21186 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.807539 21186 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:03.807554 21186 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:03.817366 21186 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:44837
I20260430 20:03:03.817453 21313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:44837 every 8 connection(s)
I20260430 20:03:03.818377 21186 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:03:03.821995 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21186
I20260430 20:03:03.822194 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:03:03.822486 10424 external_mini_cluster.cc:1468] Setting key 8a7a5371ff03eb362e6ff6f6aef7c4e5
I20260430 20:03:03.823837 20903 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.820223 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:54707 (local address 127.10.46.62:32993)
0430 20:03:03.820356 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.820360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.821156 (+ 796us) server_negotiation.cc:408] Connection header received
0430 20:03:03.821991 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.821995 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.822047 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.822148 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.823241 (+ 1093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:03.823245 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:03.823247 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 20:03:03.823259 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:03.823303 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 20:03:03.823314 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:03.823410 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:03.823635 (+ 225us) server_negotiation.cc:300] Negotiation successful
0430 20:03:03.823673 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260430 20:03:03.824590 21316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:03.820457 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:32993 (local address 127.10.46.3:54707)
0430 20:03:03.821010 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:03.821043 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:03.821759 (+ 716us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:03.822334 (+ 575us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:03.822349 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:03.822413 (+ 64us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:03.823005 (+ 592us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 20:03:03.823007 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:03.823032 (+ 25us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 20:03:03.823082 (+ 50us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:03.823480 (+ 398us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:03.823487 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:03.823491 (+ 4us) client_negotiation.cc:770] Sending connection context
0430 20:03:03.823684 (+ 193us) client_negotiation.cc:241] Negotiation successful
0430 20:03:03.823819 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260430 20:03:03.825973 21314 heartbeater.cc:344] Connected to a master server at 127.10.46.62:32993
I20260430 20:03:03.826246 21314 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:03.826678 21314 heartbeater.cc:507] Master 127.10.46.62:32993 requested a full tablet report, sending...
I20260430 20:03:03.827811 20860 ts_manager.cc:194] Registered new tserver with Master: 8b5297ea0d734177a12b39f648c1ecbf (127.10.46.3:44837)
I20260430 20:03:03.836992 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 20:03:03.840090 20903 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:03:03.839126 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.10.46.62:32993)
0430 20:03:03.839305 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:03.839310 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:03.839422 (+ 112us) server_negotiation.cc:408] Connection header received
0430 20:03:03.839575 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:03.839579 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:03.839626 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:03.839713 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:03.839952 (+ 239us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60926: BlockingRecv error: recv got EOF from 127.0.0.1:60926 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 20:03:03.840238 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20924
I20260430 20:03:03.846277 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21055
I20260430 20:03:03.851733 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21186
I20260430 20:03:03.857479 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 20833
2026-04-30T20:03:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (809 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:03:03 dist-test-slave-0bdw krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:03:03 dist-test-slave-0bdw krb5kdc[21324](info): set up 2 sockets
Apr 30 20:03:03 dist-test-slave-0bdw krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260430 20:03:05.901091 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:03:05 dist-test-slave-0bdw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:03:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:03:06.055318 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:43305
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:03:06.166344 21340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:06.166611 21340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:06.166703 21340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:06.170296 21340 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:03:06.170421 21340 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:06.170464 21340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:06.170511 21340 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:03:06.170571 21340 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:03:06.175433 21340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:06.176651 21340 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:06.177599 21340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:06.183712 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:06.183737 21345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:06.183714 21340 server_base.cc:1061] running on GCE node
W20260430 20:03:06.183737 21346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:06.184574 21340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:06.185532 21340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:06.186745 21340 hybrid_clock.cc:648] HybridClock initialized: now 1777579386186712 us; error 48 us; skew 500 ppm
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579386, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:03:06.190442 21340 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:03:06.191660 21340 webserver.cc:492] Webserver started at http://127.10.46.62:33677/ using document root <none> and password file <none>
I20260430 20:03:06.192325 21340 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:06.192400 21340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:06.192607 21340 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:06.194337 21340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "545eba592bc648198f3ee498f7be78ab"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "ceffff2457010bdfd7b5d4e6bfc331e6"
server_key_iv: "4d9002bbf635c4d5ece3590e23063179"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.194835 21340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "545eba592bc648198f3ee498f7be78ab"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "ceffff2457010bdfd7b5d4e6bfc331e6"
server_key_iv: "4d9002bbf635c4d5ece3590e23063179"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.198311 21340 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260430 20:03:06.200883 21355 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.202323 21340 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:03:06.202502 21340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "545eba592bc648198f3ee498f7be78ab"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "ceffff2457010bdfd7b5d4e6bfc331e6"
server_key_iv: "4d9002bbf635c4d5ece3590e23063179"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.202623 21340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:06.219010 21340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:06.227240 21340 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:06.227484 21340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:06.235411 21340 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:43305
I20260430 20:03:06.235447 21407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:43305 every 8 connection(s)
I20260430 20:03:06.236672 21340 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:03:06.239706 21408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:06.242067 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21340
I20260430 20:03:06.242204 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:03:06.242499 10424 external_mini_cluster.cc:1468] Setting key e4d5d50e7d2b21f5fd9ffecc95e91bcc
I20260430 20:03:06.246604 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: Bootstrap starting.
I20260430 20:03:06.249243 21408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:06.250190 21408 log.cc:826] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: Log is configured to *not* fsync() on all Append() calls
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:03:06.252556 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: No bootstrap required, opened a new log
I20260430 20:03:06.256388 21408 raft_consensus.cc:359] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } }
I20260430 20:03:06.256692 21408 raft_consensus.cc:385] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:06.256786 21408 raft_consensus.cc:740] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545eba592bc648198f3ee498f7be78ab, State: Initialized, Role: FOLLOWER
I20260430 20:03:06.257282 21408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } }
I20260430 20:03:06.257437 21408 raft_consensus.cc:399] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:03:06.257491 21408 raft_consensus.cc:493] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:03:06.257570 21408 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:06.257611 21411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:06.244125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.10.46.62:43305)
0430 20:03:06.244576 (+ 451us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:06.244584 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:06.244616 (+ 32us) server_negotiation.cc:408] Connection header received
0430 20:03:06.245431 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:06.245453 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:06.245738 (+ 285us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:06.246015 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:06.247246 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.248292 (+ 1046us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:06.249411 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.249785 (+ 374us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:06.252122 (+ 2337us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:06.252147 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:06.252162 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:03:06.252199 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:06.254607 (+ 2408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.255132 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.255139 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.255145 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.255238 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.255516 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.255520 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.255522 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.256020 (+ 498us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:06.256243 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:06.256562 (+ 319us) server_negotiation.cc:300] Negotiation successful
0430 20:03:06.256827 (+ 265us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260430 20:03:06.258775 21408 raft_consensus.cc:515] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } }
I20260430 20:03:06.259155 21408 leader_election.cc:304] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545eba592bc648198f3ee498f7be78ab; no voters:
I20260430 20:03:06.259450 21408 leader_election.cc:290] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:03:06.259572 21413 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:03:06.259832 21413 raft_consensus.cc:697] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [term 1 LEADER]: Becoming Leader. State: Replica: 545eba592bc648198f3ee498f7be78ab, State: Running, Role: LEADER
I20260430 20:03:06.260207 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } }
I20260430 20:03:06.260656 21408 sys_catalog.cc:565] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:03:06.261883 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545eba592bc648198f3ee498f7be78ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } } }
I20260430 20:03:06.262068 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [sys.catalog]: This master's current role is: LEADER
I20260430 20:03:06.261883 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545eba592bc648198f3ee498f7be78ab. Latest consensus state: current_term: 1 leader_uuid: "545eba592bc648198f3ee498f7be78ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545eba592bc648198f3ee498f7be78ab" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 43305 } } }
I20260430 20:03:06.262120 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab [sys.catalog]: This master's current role is: LEADER
I20260430 20:03:06.263043 21422 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:03:06.265803 21422 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:03:06.270668 21422 catalog_manager.cc:1357] Generated new cluster ID: c592f14d4b494df5bdb8f8e1e02872e7
I20260430 20:03:06.270750 21422 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:03:06.292340 21422 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:03:06.293982 21422 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:03:06.304877 21422 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: Generated new TSK 0
I20260430 20:03:06.305768 21422 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:03:06.368207 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:43305
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:03:06.478559 21436 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:06.478884 21436 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:06.478988 21436 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:06.482574 21436 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:06.482695 21436 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:06.482817 21436 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:03:06.487396 21436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:06.488653 21436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:06.489501 21436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:06.496351 21441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:06.496367 21444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:06.496385 21442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:06.496568 21436 server_base.cc:1061] running on GCE node
I20260430 20:03:06.497087 21436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:06.497668 21436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:06.498860 21436 hybrid_clock.cc:648] HybridClock initialized: now 1777579386498839 us; error 35 us; skew 500 ppm
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579386, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:03:06.501803 21436 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:03:06.503036 21436 webserver.cc:492] Webserver started at http://127.10.46.1:44941/ using document root <none> and password file <none>
I20260430 20:03:06.503660 21436 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:06.503739 21436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:06.503976 21436 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:06.505803 21436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "fd46c34e4d09951fce151f494de7ba09"
server_key_iv: "5340def9562ff4c8c3f5d9bc652fc8c4"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.506327 21436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "fd46c34e4d09951fce151f494de7ba09"
server_key_iv: "5340def9562ff4c8c3f5d9bc652fc8c4"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.510021 21436 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260430 20:03:06.512467 21451 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.513617 21436 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:03:06.513772 21436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "fd46c34e4d09951fce151f494de7ba09"
server_key_iv: "5340def9562ff4c8c3f5d9bc652fc8c4"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.513891 21436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:06.543485 21436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:06.546685 21436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:06.546927 21436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:06.547540 21436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:06.548630 21436 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:06.548720 21436 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.548794 21436 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:06.548846 21436 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.559435 21436 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:43981
I20260430 20:03:06.559458 21564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:43981 every 8 connection(s)
I20260430 20:03:06.560523 21436 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:03:06.565472 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21436
I20260430 20:03:06.565601 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:03:06.565856 10424 external_mini_cluster.cc:1468] Setting key d76ce9646723bf35e43f356367cd9023
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579386, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:03:06.575222 21411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:06.562508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:50171 (local address 127.10.46.62:43305)
0430 20:03:06.562666 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:06.562672 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:06.563433 (+ 761us) server_negotiation.cc:408] Connection header received
0430 20:03:06.564309 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:06.564312 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:06.564364 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:06.564447 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:06.565949 (+ 1502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.566611 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:06.567372 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.567510 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:06.570975 (+ 3465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:06.571002 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:06.571008 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:03:06.571046 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:06.572900 (+ 1854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.573595 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.573600 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.573603 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.573661 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.573971 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.573974 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.573976 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.574205 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:06.574323 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:06.574850 (+ 527us) server_negotiation.cc:300] Negotiation successful
0430 20:03:06.574992 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 20:03:06.575822 21567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:06.562829 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:43305 (local address 127.10.46.1:50171)
0430 20:03:06.563301 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:06.563336 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:06.564111 (+ 775us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:06.564791 (+ 680us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:06.564800 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:06.565241 (+ 441us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:06.565783 (+ 542us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:06.565796 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.566729 (+ 933us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:06.566732 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:06.567246 (+ 514us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:06.567255 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.567456 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:06.568779 (+ 1323us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:06.568808 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:06.570785 (+ 1977us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:06.573159 (+ 2374us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:06.573165 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:06.573177 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:06.573467 (+ 290us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:06.573769 (+ 302us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:06.573771 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:06.573773 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:06.573869 (+ 96us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:06.574330 (+ 461us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:06.574334 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:06.574569 (+ 235us) client_negotiation.cc:770] Sending connection context
0430 20:03:06.574755 (+ 186us) client_negotiation.cc:241] Negotiation successful
0430 20:03:06.574966 (+ 211us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
I20260430 20:03:06.577186 21565 heartbeater.cc:344] Connected to a master server at 127.10.46.62:43305
I20260430 20:03:06.577487 21565 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:06.578023 21565 heartbeater.cc:507] Master 127.10.46.62:43305 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
I20260430 20:03:06.579763 21372 ts_manager.cc:194] Registered new tserver with Master: c00feb06dc454b7fb5f0daf2a1eeb2c6 (127.10.46.1:43981)
I20260430 20:03:06.580991 21372 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:50171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:03:06.624435 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:43305
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:03:06.736085 21572 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:06.736346 21572 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:06.736405 21572 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:06.739920 21572 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:06.740020 21572 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:06.740108 21572 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:03:06.744599 21572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:06.745730 21572 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:06.746572 21572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:06.753335 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:06.753355 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:06.753430 21572 server_base.cc:1061] running on GCE node
W20260430 20:03:06.753331 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:06.753985 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:06.754506 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:06.755673 21572 hybrid_clock.cc:648] HybridClock initialized: now 1777579386755648 us; error 40 us; skew 500 ppm
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579386, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:03:06.758893 21572 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:03:06.760113 21572 webserver.cc:492] Webserver started at http://127.10.46.2:45891/ using document root <none> and password file <none>
I20260430 20:03:06.760712 21572 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:06.760828 21572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:06.761158 21572 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:06.763137 21572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "78ba1edeb1b5488684610b4e0e97e72f"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "1e19d4b0eb9e5830a2b530720b002ba3"
server_key_iv: "f7ec0429735a57686f69a8914cb1331c"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.763659 21572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "78ba1edeb1b5488684610b4e0e97e72f"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "1e19d4b0eb9e5830a2b530720b002ba3"
server_key_iv: "f7ec0429735a57686f69a8914cb1331c"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.767189 21572 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 20:03:06.769594 21587 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.770756 21572 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 20:03:06.770895 21572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "78ba1edeb1b5488684610b4e0e97e72f"
format_stamp: "Formatted at 2026-04-30 20:03:06 on dist-test-slave-0bdw"
server_key: "1e19d4b0eb9e5830a2b530720b002ba3"
server_key_iv: "f7ec0429735a57686f69a8914cb1331c"
server_key_version: "encryptionkey@0"
I20260430 20:03:06.771023 21572 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:06.811232 21572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:06.815330 21572 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:06.815585 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:06.816461 21572 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:06.817999 21572 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:06.818089 21572 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:06.818248 21572 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:06.818333 21572 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260430 20:03:06.832975 21572 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:36543
I20260430 20:03:06.833070 21700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:36543 every 8 connection(s)
I20260430 20:03:06.834399 21572 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:03:06.841827 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21572
I20260430 20:03:06.842015 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:03:06.842291 10424 external_mini_cluster.cc:1468] Setting key 3433fe9ac1b4721a889f1a58212a0189
Apr 30 20:03:06 dist-test-slave-0bdw krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579386, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:03:06.849545 21411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:06.836664 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.2:56433 (local address 127.10.46.62:43305)
0430 20:03:06.836834 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:06.836837 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:06.837608 (+ 771us) server_negotiation.cc:408] Connection header received
0430 20:03:06.838540 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:06.838544 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:06.838610 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:06.838713 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:06.840115 (+ 1402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.840654 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:06.841333 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.841508 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:06.844642 (+ 3134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:06.844720 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:06.844729 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:03:06.844762 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:06.846610 (+ 1848us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.847227 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.847234 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.847238 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.847298 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:06.848084 (+ 786us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:06.848088 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:06.848090 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:06.848329 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:06.848452 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:06.849139 (+ 687us) server_negotiation.cc:300] Negotiation successful
0430 20:03:06.849293 (+ 154us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260430 20:03:06.850069 21703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:06.837028 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:43305 (local address 127.10.46.2:56433)
0430 20:03:06.837467 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:06.837501 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:06.838333 (+ 832us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:06.838876 (+ 543us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:06.838885 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:06.839312 (+ 427us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:06.839860 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:06.839879 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.840791 (+ 912us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:06.840795 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:06.841204 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:06.841212 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:06.841439 (+ 227us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:06.842686 (+ 1247us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:06.842712 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:06.844468 (+ 1756us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:06.846781 (+ 2313us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:06.846787 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:06.846798 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:06.847080 (+ 282us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:06.847519 (+ 439us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:06.847525 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:06.847527 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:06.847871 (+ 344us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:06.848501 (+ 630us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:06.848507 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:06.848804 (+ 297us) client_negotiation.cc:770] Sending connection context
0430 20:03:06.849026 (+ 222us) client_negotiation.cc:241] Negotiation successful
0430 20:03:06.849250 (+ 224us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260430 20:03:06.851253 21701 heartbeater.cc:344] Connected to a master server at 127.10.46.62:43305
I20260430 20:03:06.851503 21701 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.10.46.3@KRBTEST.COM; defaulting to no policy
I20260430 20:03:06.852164 21701 heartbeater.cc:507] Master 127.10.46.62:43305 requested a full tablet report, sending...
I20260430 20:03:06.853331 21372 ts_manager.cc:194] Registered new tserver with Master: 78ba1edeb1b5488684610b4e0e97e72f (127.10.46.2:36543)
I20260430 20:03:06.854000 21372 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.2@KRBTEST.COM'} at 127.10.46.2:56433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:03:06.901800 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.3:0
--local_ip_for_outbound_sockets=127.10.46.3
--webserver_interface=127.10.46.3
--webserver_port=0
--tserver_master_addrs=127.10.46.62:43305
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:03:07.019165 21708 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:03:07.019433 21708 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:03:07.019497 21708 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:03:07.023133 21708 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:03:07.023223 21708 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:03:07.023308 21708 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.3
I20260430 20:03:07.028383 21708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.46.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.3
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:03:07.029515 21708 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:03:07.030359 21708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:03:07.037915 21716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:07.037992 21714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:03:07.038017 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:03:07.038455 21708 server_base.cc:1061] running on GCE node
I20260430 20:03:07.039476 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:03:07.040381 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:03:07.041611 21708 hybrid_clock.cc:648] HybridClock initialized: now 1777579387041581 us; error 51 us; skew 500 ppm
Apr 30 20:03:07 dist-test-slave-0bdw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579387, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:03:07.045300 21708 init.cc:377] Logged in from keytab as kudu/127.10.46.3@KRBTEST.COM (short username kudu)
I20260430 20:03:07.046625 21708 webserver.cc:492] Webserver started at http://127.10.46.3:35949/ using document root <none> and password file <none>
I20260430 20:03:07.047258 21708 fs_manager.cc:362] Metadata directory not provided
I20260430 20:03:07.047338 21708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:03:07.047545 21708 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 20:03:07.049410 21708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/instance:
uuid: "546beaed31c04767a300bc8bc1bf7b04"
format_stamp: "Formatted at 2026-04-30 20:03:07 on dist-test-slave-0bdw"
server_key: "fa3c36322f5a03f4339cda476585482a"
server_key_iv: "5b744478632931e18857bf85cc6189d3"
server_key_version: "encryptionkey@0"
I20260430 20:03:07.049940 21708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance:
uuid: "546beaed31c04767a300bc8bc1bf7b04"
format_stamp: "Formatted at 2026-04-30 20:03:07 on dist-test-slave-0bdw"
server_key: "fa3c36322f5a03f4339cda476585482a"
server_key_iv: "5b744478632931e18857bf85cc6189d3"
server_key_version: "encryptionkey@0"
I20260430 20:03:07.053543 21708 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 20:03:07.056255 21723 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:07.057371 21708 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 20:03:07.057480 21708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
uuid: "546beaed31c04767a300bc8bc1bf7b04"
format_stamp: "Formatted at 2026-04-30 20:03:07 on dist-test-slave-0bdw"
server_key: "fa3c36322f5a03f4339cda476585482a"
server_key_iv: "5b744478632931e18857bf85cc6189d3"
server_key_version: "encryptionkey@0"
I20260430 20:03:07.057572 21708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:03:07.078392 21708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:03:07.081750 21708 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:03:07.081929 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:03:07.082466 21708 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:03:07.083360 21708 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:03:07.083410 21708 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:07.083451 21708 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:03:07.083467 21708 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:03:07.093174 21708 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.3:39955
I20260430 20:03:07.093190 21836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.3:39955 every 8 connection(s)
I20260430 20:03:07.094329 21708 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/data/info.pb
I20260430 20:03:07.098920 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 21708
I20260430 20:03:07.099090 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/instance
I20260430 20:03:07.099385 10424 external_mini_cluster.cc:1468] Setting key d0161c18057029de19b6f06d4faf6200
Apr 30 20:03:07 dist-test-slave-0bdw krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579387, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.3@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:03:07.109752 21411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.096469 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.3:50379 (local address 127.10.46.62:43305)
0430 20:03:07.096619 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.096623 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.097380 (+ 757us) server_negotiation.cc:408] Connection header received
0430 20:03:07.098231 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.098235 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.098286 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.098370 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:07.100090 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.100811 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.101742 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.101932 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.105670 (+ 3738us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:07.105692 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:07.105695 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:03:07.105726 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:07.107087 (+ 1361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:07.108130 (+ 1043us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:07.108133 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:07.108135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:07.108189 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:07.108589 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:07.108592 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:07.108594 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:07.108761 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:07.108858 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.109501 (+ 643us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.109616 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 20:03:07.110342 21839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.096792 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:43305 (local address 127.10.46.3:50379)
0430 20:03:07.097230 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.097265 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.098031 (+ 766us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.098558 (+ 527us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.098567 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.099202 (+ 635us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:03:07.099924 (+ 722us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.099939 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.101012 (+ 1073us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.101016 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.101596 (+ 580us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.101607 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.101937 (+ 330us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.103583 (+ 1646us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:03:07.103607 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:03:07.105467 (+ 1860us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:03:07.107342 (+ 1875us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:07.107352 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:07.107376 (+ 24us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:07.107995 (+ 619us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:07.108312 (+ 317us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:03:07.108316 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:03:07.108317 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:03:07.108439 (+ 122us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:03:07.108913 (+ 474us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:03:07.108919 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:03:07.109166 (+ 247us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.109356 (+ 190us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.109569 (+ 213us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260430 20:03:07.111492 21837 heartbeater.cc:344] Connected to a master server at 127.10.46.62:43305
I20260430 20:03:07.111826 21837 heartbeater.cc:461] Registering TS with master...
I20260430 20:03:07.112450 21837 heartbeater.cc:507] Master 127.10.46.62:43305 requested a full tablet report, sending...
I20260430 20:03:07.113598 21371 ts_manager.cc:194] Registered new tserver with Master: 546beaed31c04767a300bc8bc1bf7b04 (127.10.46.3:39955)
I20260430 20:03:07.114238 21371 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.3@KRBTEST.COM'} at 127.10.46.3:50379
I20260430 20:03:07.126083 10424 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 20:03:07.136986 21411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.128528 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.10.46.62:43305)
0430 20:03:07.128755 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.128760 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.128817 (+ 57us) server_negotiation.cc:408] Connection header received
0430 20:03:07.129048 (+ 231us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.129051 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.129131 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.129228 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:03:07.130332 (+ 1104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.130925 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.131735 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.132015 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.133148 (+ 1133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:03:07.133175 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:03:07.133177 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:03:07.133217 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:03:07.134732 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:07.135444 (+ 712us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:07.135447 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:07.135449 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:07.135503 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:03:07.135918 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:03:07.135921 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:03:07.135922 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:03:07.136133 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:03:07.136272 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.136544 (+ 272us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.136809 (+ 265us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260430 20:03:07.141646 21371 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 20:03:07.144379 21371 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 20:03:07.160830 21852 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.155633 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:39955 (local address 127.0.0.1:43134)
0430 20:03:07.156683 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.156708 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.156825 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.157481 (+ 656us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.157484 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.157499 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:03:07.158091 (+ 592us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.158099 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159367 (+ 1268us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.159373 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.160299 (+ 926us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.160308 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.160426 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.160493 (+ 67us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.160590 (+ 97us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.160668 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":190,"threads_started":1}
I20260430 20:03:07.161473 21851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.155903 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.10.46.3:39955)
0430 20:03:07.156756 (+ 853us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.156764 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.156789 (+ 25us) server_negotiation.cc:408] Connection header received
0430 20:03:07.157026 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.157032 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.157226 (+ 194us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.157509 (+ 283us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:03:07.158219 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159243 (+ 1024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.160470 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.161017 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.161165 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.161237 (+ 72us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.161326 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":172,"threads_started":1}
I20260430 20:03:07.161844 21850 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.156119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:36543 (local address 127.0.0.1:35252)
0430 20:03:07.156962 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.156978 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.157061 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.157624 (+ 563us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.157626 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.157637 (+ 11us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:03:07.157823 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.157844 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159730 (+ 1886us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.159734 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.160591 (+ 857us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.160600 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.160721 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.160737 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.160787 (+ 50us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.160839 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":260,"threads_started":1}
I20260430 20:03:07.162420 21849 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.154965 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.10.46.1:43981)
0430 20:03:07.156168 (+ 1203us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.156177 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.156208 (+ 31us) server_negotiation.cc:408] Connection header received
0430 20:03:07.156792 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.156800 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.157055 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.157207 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:03:07.158042 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159536 (+ 1494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.160671 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.161362 (+ 691us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.161490 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.162105 (+ 615us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.162209 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":176,"threads_started":1}
I20260430 20:03:07.161970 21848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.154617 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:43981 (local address 127.0.0.1:46866)
0430 20:03:07.155607 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.155659 (+ 52us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.156495 (+ 836us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.157380 (+ 885us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.157384 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.157403 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:03:07.157788 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.157796 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159669 (+ 1873us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.159673 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.160498 (+ 825us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.160505 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.161636 (+ 1131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.161652 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.161770 (+ 118us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.161841 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":93,"threads_started":1}
I20260430 20:03:07.163407 21853 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.156337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.10.46.2:36543)
0430 20:03:07.157045 (+ 708us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.157051 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.157073 (+ 22us) server_negotiation.cc:408] Connection header received
0430 20:03:07.157155 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.157160 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.157415 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.157574 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:03:07.158076 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.159535 (+ 1459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.161240 (+ 1705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.162944 (+ 1704us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.163039 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.163109 (+ 70us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.163266 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":274,"threads_started":1}
I20260430 20:03:07.164868 21771 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 (DEFAULT_TABLE table=test-table [id=a183b77f4cdb486b838e7d4f8249d3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:03:07.165299 21498 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 (DEFAULT_TABLE table=test-table [id=a183b77f4cdb486b838e7d4f8249d3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:03:07.165786 21635 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 (DEFAULT_TABLE table=test-table [id=a183b77f4cdb486b838e7d4f8249d3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 20:03:07.166412 21771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3f186f6ec34ce9be73bffa4d85c430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:07.166425 21498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3f186f6ec34ce9be73bffa4d85c430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:07.166661 21635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3f186f6ec34ce9be73bffa4d85c430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:03:07.172178 21854 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Bootstrap starting.
I20260430 20:03:07.172963 21855 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: Bootstrap starting.
I20260430 20:03:07.174312 21854 tablet_bootstrap.cc:654] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:07.174576 21855 tablet_bootstrap.cc:654] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:07.175374 21855 log.cc:826] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: Log is configured to *not* fsync() on all Append() calls
I20260430 20:03:07.175822 21854 log.cc:826] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Log is configured to *not* fsync() on all Append() calls
I20260430 20:03:07.175964 21856 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Bootstrap starting.
I20260430 20:03:07.178226 21855 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: No bootstrap required, opened a new log
I20260430 20:03:07.178320 21854 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: No bootstrap required, opened a new log
I20260430 20:03:07.178458 21855 ts_tablet_manager.cc:1403] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260430 20:03:07.178504 21854 ts_tablet_manager.cc:1403] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260430 20:03:07.178843 21856 tablet_bootstrap.cc:654] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Neither blocks nor log segments found. Creating new log.
I20260430 20:03:07.179744 21856 log.cc:826] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Log is configured to *not* fsync() on all Append() calls
I20260430 20:03:07.181373 21855 raft_consensus.cc:359] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.181592 21855 raft_consensus.cc:385] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:07.181655 21855 raft_consensus.cc:740] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00feb06dc454b7fb5f0daf2a1eeb2c6, State: Initialized, Role: FOLLOWER
I20260430 20:03:07.181958 21856 tablet_bootstrap.cc:492] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: No bootstrap required, opened a new log
I20260430 20:03:07.182129 21855 consensus_queue.cc:260] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.182181 21856 ts_tablet_manager.cc:1403] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260430 20:03:07.182199 21854 raft_consensus.cc:359] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.182404 21854 raft_consensus.cc:385] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:07.182466 21854 raft_consensus.cc:740] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546beaed31c04767a300bc8bc1bf7b04, State: Initialized, Role: FOLLOWER
I20260430 20:03:07.182840 21854 consensus_queue.cc:260] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.183856 21854 ts_tablet_manager.cc:1434] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260430 20:03:07.183820 21855 ts_tablet_manager.cc:1434] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 20:03:07.184409 21837 heartbeater.cc:499] Master 127.10.46.62:43305 was elected leader, sending a full tablet report...
I20260430 20:03:07.185742 21856 raft_consensus.cc:359] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.185920 21565 heartbeater.cc:499] Master 127.10.46.62:43305 was elected leader, sending a full tablet report...
I20260430 20:03:07.186014 21856 raft_consensus.cc:385] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:03:07.186076 21856 raft_consensus.cc:740] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ba1edeb1b5488684610b4e0e97e72f, State: Initialized, Role: FOLLOWER
I20260430 20:03:07.186538 21856 consensus_queue.cc:260] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.187415 21701 heartbeater.cc:499] Master 127.10.46.62:43305 was elected leader, sending a full tablet report...
I20260430 20:03:07.187556 21856 ts_tablet_manager.cc:1434] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260430 20:03:07.206398 21860 raft_consensus.cc:493] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 20:03:07.206648 21860 raft_consensus.cc:515] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.207942 21860 leader_election.cc:290] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 546beaed31c04767a300bc8bc1bf7b04 (127.10.46.3:39955), 78ba1edeb1b5488684610b4e0e97e72f (127.10.46.2:36543)
I20260430 20:03:07.213289 21864 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.208705 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.2:36543 (local address 127.10.46.1:58535)
0430 20:03:07.209041 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.209060 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.209204 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.209585 (+ 381us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.209587 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.209601 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:03:07.209976 (+ 375us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.209983 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.211349 (+ 1366us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.211356 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.212749 (+ 1393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.212761 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.212941 (+ 180us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.212961 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.213018 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.213098 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
I20260430 20:03:07.213289 21863 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.208160 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.3:39955 (local address 127.10.46.1:36949)
0430 20:03:07.208694 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.208718 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0430 20:03:07.208930 (+ 212us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:03:07.209362 (+ 432us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:03:07.209364 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:03:07.209382 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 20:03:07.210052 (+ 670us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.210061 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.211349 (+ 1288us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.211356 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:03:07.212746 (+ 1390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:03:07.212761 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.212941 (+ 180us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.212961 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 20:03:07.213018 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 20:03:07.213085 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":199,"threads_started":1}
I20260430 20:03:07.213759 21851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.208552 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:36949 (local address 127.10.46.3:39955)
0430 20:03:07.208745 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.208756 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.208829 (+ 73us) server_negotiation.cc:408] Connection header received
0430 20:03:07.209138 (+ 309us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.209141 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.209233 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.209333 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:03:07.210258 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.211113 (+ 855us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.212957 (+ 1844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.213471 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.213512 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.213571 (+ 59us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.213626 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260430 20:03:07.213757 21853 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.209108 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:58535 (local address 127.10.46.2:36543)
0430 20:03:07.209323 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.209328 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.209343 (+ 15us) server_negotiation.cc:408] Connection header received
0430 20:03:07.209405 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.209408 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.209464 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.209543 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 20:03:07.210255 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.211100 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.212948 (+ 1848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.213471 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.213512 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.213571 (+ 59us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.213626 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260430 20:03:07.214840 21791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3f186f6ec34ce9be73bffa4d85c430" candidate_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546beaed31c04767a300bc8bc1bf7b04" is_pre_election: true
I20260430 20:03:07.214924 21655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3f186f6ec34ce9be73bffa4d85c430" candidate_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ba1edeb1b5488684610b4e0e97e72f" is_pre_election: true
I20260430 20:03:07.215262 21655 raft_consensus.cc:2468] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00feb06dc454b7fb5f0daf2a1eeb2c6 in term 0.
I20260430 20:03:07.215263 21791 raft_consensus.cc:2468] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00feb06dc454b7fb5f0daf2a1eeb2c6 in term 0.
I20260430 20:03:07.216045 21453 leader_election.cc:304] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78ba1edeb1b5488684610b4e0e97e72f, c00feb06dc454b7fb5f0daf2a1eeb2c6; no voters:
I20260430 20:03:07.216362 21860 raft_consensus.cc:2804] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 20:03:07.216449 21860 raft_consensus.cc:493] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 20:03:07.216492 21860 raft_consensus.cc:3060] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:07.217515 21860 raft_consensus.cc:515] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.218012 21860 leader_election.cc:290] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [CANDIDATE]: Term 1 election: Requested vote from peers 546beaed31c04767a300bc8bc1bf7b04 (127.10.46.3:39955), 78ba1edeb1b5488684610b4e0e97e72f (127.10.46.2:36543)
I20260430 20:03:07.218413 21791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3f186f6ec34ce9be73bffa4d85c430" candidate_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546beaed31c04767a300bc8bc1bf7b04"
I20260430 20:03:07.218483 21655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3f186f6ec34ce9be73bffa4d85c430" candidate_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ba1edeb1b5488684610b4e0e97e72f"
I20260430 20:03:07.218583 21791 raft_consensus.cc:3060] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:07.218621 21655 raft_consensus.cc:3060] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:03:07.219951 21791 raft_consensus.cc:2468] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00feb06dc454b7fb5f0daf2a1eeb2c6 in term 1.
I20260430 20:03:07.219951 21655 raft_consensus.cc:2468] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00feb06dc454b7fb5f0daf2a1eeb2c6 in term 1.
I20260430 20:03:07.220281 21453 leader_election.cc:304] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78ba1edeb1b5488684610b4e0e97e72f, c00feb06dc454b7fb5f0daf2a1eeb2c6; no voters:
I20260430 20:03:07.220463 21860 raft_consensus.cc:2804] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:03:07.220734 21860 raft_consensus.cc:697] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 1 LEADER]: Becoming Leader. State: Replica: c00feb06dc454b7fb5f0daf2a1eeb2c6, State: Running, Role: LEADER
I20260430 20:03:07.221212 21860 consensus_queue.cc:237] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } }
I20260430 20:03:07.224790 21372 catalog_manager.cc:5671] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c00feb06dc454b7fb5f0daf2a1eeb2c6 (127.10.46.1). New cstate: current_term: 1 leader_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00feb06dc454b7fb5f0daf2a1eeb2c6" member_type: VOTER last_known_addr { host: "127.10.46.1" port: 43981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 } health_report { overall_health: UNKNOWN } } }
I20260430 20:03:07.244678 21849 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.241276 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.10.46.1:43981)
0430 20:03:07.241437 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 20:03:07.241444 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 20:03:07.241463 (+ 19us) server_negotiation.cc:408] Connection header received
0430 20:03:07.241567 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:03:07.241570 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:03:07.241625 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:03:07.241734 (+ 109us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 20:03:07.242186 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.242719 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:03:07.243462 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:03:07.243674 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:03:07.243761 (+ 87us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 20:03:07.244221 (+ 460us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 20:03:07.244348 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0430 20:03:07.244451 (+ 103us) server_negotiation.cc:300] Negotiation successful
0430 20:03:07.244522 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 20:03:07.251714 21791 raft_consensus.cc:1275] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 1 FOLLOWER]: Refusing update from remote peer c00feb06dc454b7fb5f0daf2a1eeb2c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:03:07.251714 21655 raft_consensus.cc:1275] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 1 FOLLOWER]: Refusing update from remote peer c00feb06dc454b7fb5f0daf2a1eeb2c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 20:03:07.252589 21865 consensus_queue.cc:1048] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ba1edeb1b5488684610b4e0e97e72f" member_type: VOTER last_known_addr { host: "127.10.46.2" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:03:07.252993 21860 consensus_queue.cc:1048] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "546beaed31c04767a300bc8bc1bf7b04" member_type: VOTER last_known_addr { host: "127.10.46.3" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 20:03:07.261989 21869 mvcc.cc:204] Tried to move back new op lower bound from 7280965170174304256 to 7280965170061172736. Current Snapshot: MvccSnapshot[applied={T|T < 7280965170174304256}]
I20260430 20:03:07.273119 21372 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:03:07.273347 21372 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 20:03:07.276090 21372 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 545eba592bc648198f3ee498f7be78ab: Sending DeleteTablet for 3 replicas of tablet 7f3f186f6ec34ce9be73bffa4d85c430
I20260430 20:03:07.277029 21635 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:03:07 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:35252
I20260430 20:03:07.277033 21771 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:03:07 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:43134
I20260430 20:03:07.277040 21498 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3f186f6ec34ce9be73bffa4d85c430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 20:03:07 UTC) from {username='kudu', principal='kudu/127.10.46.62@KRBTEST.COM'} at 127.0.0.1:46866
I20260430 20:03:07.277563 21878 tablet_replica.cc:333] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: stopping tablet replica
I20260430 20:03:07.277853 21879 tablet_replica.cc:333] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6: stopping tablet replica
I20260430 20:03:07.277863 21878 raft_consensus.cc:2243] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:03:07.278162 21879 raft_consensus.cc:2243] T 7f3f186f6ec34ce9be73bffa4d85c430 P c00feb06dc454b7fb5f0daf2a1eeb2c6 [term 1 LEADER]: Raft consensus shutting down.
I20260430 20:03:07.278236 21878 raft_consensus.cc:2272] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:03:07.278422 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21436
I20260430 20:03:07.279379 21880 tablet_replica.cc:333] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: stopping tablet replica
I20260430 20:03:07.279640 21880 raft_consensus.cc:2243] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 20:03:07.280218 21880 raft_consensus.cc:2272] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 20:03:07.280344 21878 ts_tablet_manager.cc:1916] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:03:07.282620 21880 ts_tablet_manager.cc:1916] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 20:03:07.284260 21878 ts_tablet_manager.cc:1929] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:03:07.284377 21878 log.cc:1199] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-1/wal/wals/7f3f186f6ec34ce9be73bffa4d85c430
I20260430 20:03:07.284698 21878 ts_tablet_manager.cc:1950] T 7f3f186f6ec34ce9be73bffa4d85c430 P 78ba1edeb1b5488684610b4e0e97e72f: Deleting consensus metadata
I20260430 20:03:07.285084 21880 ts_tablet_manager.cc:1929] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 20:03:07.285202 21880 log.cc:1199] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Deleting WAL directory at /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777579284342945-10424-0/minicluster-data/ts-2/wal/wals/7f3f186f6ec34ce9be73bffa4d85c430
I20260430 20:03:07.285491 21880 ts_tablet_manager.cc:1950] T 7f3f186f6ec34ce9be73bffa4d85c430 P 546beaed31c04767a300bc8bc1bf7b04: Deleting consensus metadata
I20260430 20:03:07.286823 21357 catalog_manager.cc:5002] TS 546beaed31c04767a300bc8bc1bf7b04 (127.10.46.3:39955): tablet 7f3f186f6ec34ce9be73bffa4d85c430 (table test-table [id=a183b77f4cdb486b838e7d4f8249d3f5]) successfully deleted
I20260430 20:03:07.287132 21357 catalog_manager.cc:5002] TS 78ba1edeb1b5488684610b4e0e97e72f (127.10.46.2:36543): tablet 7f3f186f6ec34ce9be73bffa4d85c430 (table test-table [id=a183b77f4cdb486b838e7d4f8249d3f5]) successfully deleted
W20260430 20:03:07.288595 21357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.46.1:43981 (error 108)
I20260430 20:03:07.289397 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21572
I20260430 20:03:07.289722 21850 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:03:07.289402 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.1:43981 (local address 127.0.0.1:46874)
0430 20:03:07.289563 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0430 20:03:07.289623 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260430 20:03:07.291244 21357 catalog_manager.cc:4729] TS c00feb06dc454b7fb5f0daf2a1eeb2c6 (127.10.46.1:43981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f3f186f6ec34ce9be73bffa4d85c430: Network error: Client connection negotiation failed: client connection to 127.10.46.1:43981: connect: Connection refused (error 111)
I20260430 20:03:07.296224 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21708
I20260430 20:03:07.302390 10424 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu with pid 21340
2026-04-30T20:03:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3446 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 20:03:07 dist-test-slave-0bdw krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 20:03:07 dist-test-slave-0bdw krb5kdc[21884](info): set up 2 sockets
Apr 30 20:03:07 dist-test-slave-0bdw krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260430 20:03:09.370025 10424 mini_kdc.cc:121] Time spent starting KDC: real 2.044s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/test-user.keytab.
Apr 30 20:03:09 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T20:03:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T20:03:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.46.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.46.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangeradmin_127.10.46.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangeradmin_127.10.46.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.46.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.46.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangerlookup_127.10.46.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangerlookup_127.10.46.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.46.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/HTTP_127.10.46.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/HTTP_127.10.46.20@KRBTEST.COM.keytab.
I20260430 20:03:09.560418 10424 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqmdCRr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/postgres -l logfile start
2026-04-30 20:03:12.679 UTC [21915] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-30 20:03:12.680 UTC [21915] LOG: listening on IPv4 address "127.10.46.20", port 51957
2026-04-30 20:03:12.684 UTC [21915] LOG: listening on Unix socket "/tmp/.s.PGSQL.51957"
2026-04-30 20:03:12.690 UTC [21920] LOG: database system was shut down at 2026-04-30 20:03:10 UTC
2026-04-30 20:03:12.694 UTC [21915] LOG: database system is ready to accept connections
I20260430 20:03:14.669595 10424 mini_postgres.cc:96] Postgres bound to 51957
2026-04-30 20:03:14.676 UTC [21926] FATAL: database "slave" does not exist
127.10.46.20:51957 - accepting connections
I20260430 20:03:14.676693 10424 mini_ranger.cc:161] Starting Ranger...
I20260430 20:03:14.696281 10424 mini_ranger.cc:84] Created miniranger Postgres user
I20260430 20:03:14.773376 10424 mini_ranger.cc:87] Created ranger Postgres database
I20260430 20:03:14.773497 10424 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin
2026-04-30 20:03:15,075 [I] DB FLAVOR :POSTGRES
2026-04-30 20:03:15,076 [I] --------- Verifying Ranger DB connection ---------
2026-04-30 20:03:15,076 [I] Checking connection..
2026-04-30 20:03:15,076 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-30 20:03:15,439 [I] Checking connection passed.
2026-04-30 20:03:15,439 [I] --------- Verifying version history table ---------
2026-04-30 20:03:15,440 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 20:03:15,780 [I] Table x_db_version_h does not exist in database ranger
2026-04-30 20:03:15,781 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 20:03:16,148 [I] Table x_db_version_h does not exist in database ranger
2026-04-30 20:03:16,149 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-30 20:03:16,149 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-30 20:03:16.480 UTC [22039] WARNING: there is no transaction in progress
2026-04-30 20:03:16,498 [I] create_dbversion_catalog.sql file imported successfully
2026-04-30 20:03:16,498 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 20:03:16,897 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-30 20:03:16,898 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-30 20:03:16,898 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-30 20:03:17,267 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-30 20:03:17,633 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0bdw.c.gcp-upstream.internal','N') ;"
2026-04-30 20:03:18,010 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-30 20:03:18,388 [I] Table x_portal_user does not exist in database ranger
2026-04-30 20:03:18,389 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-30 20:03:18,741 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-30 20:03:18,741 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-30 20:03:19,121 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-30 20:03:19,495 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-30 20:03:19,496 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-30 20:03:20.525 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.539 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.553 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.565 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.579 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.593 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.657 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.665 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.681 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.696 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.709 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.722 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.734 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.741 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.747 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:20.754 UTC [22238] WARNING: there is no transaction in progress
2026-04-30 20:03:21,371 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-30 20:03:21,371 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-30 20:03:21,694 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-30 20:03:21,695 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-30 20:03:22,074 [I] Table x_portal_user already exists in database 'ranger'
2026-04-30 20:03:22,074 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-30 20:03:22,414 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-30 20:03:22,415 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-30 20:03:22,735 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-30 20:03:23,059 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0bdw.c.gcp-upstream.internal';"
2026-04-30 20:03:23,424 [I] CORE_DB_SCHEMA import status has been updated
2026-04-30 20:03:23,425 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-30 20:03:23,764 [I] DB_PATCHES have already been applied
I20260430 20:03:23.770699 10424 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskqmdCRr/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260430 20:03:23.770771 10424 mini_ranger.cc:193] Using host: 127.10.46.20
I20260430 20:03:23.773905 10424 mini_ranger.cc:239] Ranger admin URL: http://127.10.46.20:56173
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin/ews/webapp
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-admin/ews/webapp].
Apr 30, 2026 8:03:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260430 20:03:26.163383 10424 mini_ranger.cc:160] Time spent starting Ranger: real 11.487s user 0.004s sys 0.008s
Apr 30, 2026 8:03:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 30, 2026 8:03:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XUserREST
Apr 30, 2026 8:03:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 30, 2026 8:03:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 30, 2026 8:03:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260430 20:03:52.421665 10424 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.46.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.46.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangerkms_127.10.46.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.46.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/rangerkms_127.10.46.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.46.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.46.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/HTTP_127.10.46.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.46.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/HTTP_127.10.46.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 30 20:03:52 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579432, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260430 20:03:52.534309 10424 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260430 20:03:52.547232 10424 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260430 20:03:52.684640 10424 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260430 20:03:52.684747 10424 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms
I20260430 20:03:52.684787 10424 mini_ranger_kms.cc:227] Using postgres at 127.10.46.20:51957
2026-04-30 20:03:52,921 [I] DB FLAVOR :POSTGRES
2026-04-30 20:03:52,922 [I] --------- Verifying Ranger DB connection ---------
2026-04-30 20:03:52,922 [I] Checking connection
2026-04-30 20:03:52,922 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-30 20:03:53,360 [I] connection success
2026-04-30 20:03:53,361 [I] --------- Verifying Ranger DB tables ---------
2026-04-30 20:03:53,361 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-30 20:03:53,361 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-30 20:03:53,696 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-30 20:03:53,696 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-30 20:03:53,697 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-30 20:03:53,697 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.46.20:51957/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-30 20:03:54,087 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260430 20:03:54.718518 10424 mini_ranger_kms.cc:326] Created kms service
I20260430 20:03:54.957742 10424 mini_ranger_kms.cc:342] Created kudu user
I20260430 20:03:55.006691 10424 mini_ranger_kms.cc:359] Created rangerkms user
Apr 30 20:03:55 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579432, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
I20260430 20:03:55.420208 10424 mini_ranger_kms.cc:400] Added ranger policy
I20260430 20:03:55.420420 10424 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqmdCRr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqmdCRr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260430 20:03:55.420454 10424 mini_ranger_kms.cc:240] Using host: 127.10.46.20
I20260430 20:03:55.424178 10424 mini_ranger_kms.cc:292] Ranger KMS PID: 22538
I20260430 20:03:55.424293 10424 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.10.46.20:34783
Apr 30, 2026 8:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 30, 2026 8:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 30, 2026 8:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 30, 2026 8:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 30, 2026 8:03:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-kms/ews/webapp].
Apr 30, 2026 8:03:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260430 20:03:57.769870 10424 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.236s user 0.002s sys 0.204s
I20260430 20:03:57.770004 10424 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260430 20:03:57.770082 10424 mini_ranger_kms.cc:417] 127.10.46.20:34783/kms/v1/keys
Apr 30 20:03:58 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777579438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.46.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 30 20:03:58 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777579438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.46.20@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
Apr 30 20:03:59 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777579438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.46.20@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
Apr 30 20:04:02 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:04:02.216814 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37697
--webserver_interface=127.10.46.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.46.62:37697
--ranger_config_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.46.20:34783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:04:02.326997 22606 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:04:02.327255 22606 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:04:02.327309 22606 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:04:02.330876 22606 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 20:04:02.330972 22606 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:04:02.330996 22606 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:04:02.331015 22606 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 20:04:02.331033 22606 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 20:04:02.336053 22606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.46.20:34783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.46.62:37697
--ranger_config_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.46.62:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.46.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:04:02.337297 22606 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:04:02.338184 22606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:04:02.344378 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:04:02.344445 22606 server_base.cc:1061] running on GCE node
W20260430 20:04:02.344388 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:04:02.344378 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:04:02.345189 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:04:02.346156 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:04:02.347350 22606 hybrid_clock.cc:648] HybridClock initialized: now 1777579442347319 us; error 41 us; skew 500 ppm
Apr 30 20:04:02 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579442, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:04:02.350488 22606 init.cc:377] Logged in from keytab as kudu/127.10.46.62@KRBTEST.COM (short username kudu)
I20260430 20:04:02.351645 22606 webserver.cc:492] Webserver started at http://127.10.46.62:44389/ using document root <none> and password file <none>
I20260430 20:04:02.352276 22606 fs_manager.cc:362] Metadata directory not provided
I20260430 20:04:02.352353 22606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:04:02.352579 22606 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 20:04:02 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579442, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.62@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
I20260430 20:04:02.404276 22606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data/instance:
uuid: "730b3cd235c04dfda4e82dd31bb10055"
format_stamp: "Formatted at 2026-04-30 20:04:02 on dist-test-slave-0bdw"
server_key: "07833cc3417e6e0daf755043c112b69e"
server_key_iv: "f9aef3176e1f5a3c36094ad3563c8553"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:02.404974 22606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal/instance:
uuid: "730b3cd235c04dfda4e82dd31bb10055"
format_stamp: "Formatted at 2026-04-30 20:04:02 on dist-test-slave-0bdw"
server_key: "07833cc3417e6e0daf755043c112b69e"
server_key_iv: "f9aef3176e1f5a3c36094ad3563c8553"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:02.409934 22606 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260430 20:04:02.429281 22621 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:02.430559 22606 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 20:04:02.430737 22606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal
uuid: "730b3cd235c04dfda4e82dd31bb10055"
format_stamp: "Formatted at 2026-04-30 20:04:02 on dist-test-slave-0bdw"
server_key: "07833cc3417e6e0daf755043c112b69e"
server_key_iv: "f9aef3176e1f5a3c36094ad3563c8553"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:02.430866 22606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:04:02.446763 22606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:04:02.455476 22606 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:04:02.455718 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:04:02.463727 22606 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.62:37697
I20260430 20:04:02.463725 22673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.62:37697 every 8 connection(s)
I20260430 20:04:02.465131 22606 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/data/info.pb
I20260430 20:04:02.468750 22674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 20:04:02.473670 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055: Bootstrap starting.
I20260430 20:04:02.474959 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 22606
I20260430 20:04:02.475101 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/wal/instance
I20260430 20:04:02.475366 10424 external_mini_cluster.cc:1468] Setting key 2da916e96b544427855f7a69eb389cb4
I20260430 20:04:02.476466 22674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055: Neither blocks nor log segments found. Creating new log.
I20260430 20:04:02.477243 22674 log.cc:826] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055: Log is configured to *not* fsync() on all Append() calls
I20260430 20:04:02.479382 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055: No bootstrap required, opened a new log
I20260430 20:04:02.483438 22674 raft_consensus.cc:359] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } }
Apr 30 20:04:02 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:04:02.483639 22674 raft_consensus.cc:385] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 20:04:02.483675 22674 raft_consensus.cc:740] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730b3cd235c04dfda4e82dd31bb10055, State: Initialized, Role: FOLLOWER
I20260430 20:04:02.484208 22674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } }
I20260430 20:04:02.484331 22674 raft_consensus.cc:399] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 20:04:02.484444 22674 raft_consensus.cc:493] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 20:04:02.484541 22674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 0 FOLLOWER]: Advancing to term 1
I20260430 20:04:02.485661 22674 raft_consensus.cc:515] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } }
I20260430 20:04:02.486038 22674 leader_election.cc:304] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730b3cd235c04dfda4e82dd31bb10055; no voters:
I20260430 20:04:02.486377 22674 leader_election.cc:290] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 20:04:02.486481 22679 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 20:04:02.486672 22679 raft_consensus.cc:697] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [term 1 LEADER]: Becoming Leader. State: Replica: 730b3cd235c04dfda4e82dd31bb10055, State: Running, Role: LEADER
I20260430 20:04:02.487018 22679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } }
I20260430 20:04:02.487735 22674 sys_catalog.cc:565] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 20:04:02.488754 22681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730b3cd235c04dfda4e82dd31bb10055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } } }
I20260430 20:04:02.489097 22681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [sys.catalog]: This master's current role is: LEADER
I20260430 20:04:02.489319 22680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730b3cd235c04dfda4e82dd31bb10055. Latest consensus state: current_term: 1 leader_uuid: "730b3cd235c04dfda4e82dd31bb10055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b3cd235c04dfda4e82dd31bb10055" member_type: VOTER last_known_addr { host: "127.10.46.62" port: 37697 } } }
I20260430 20:04:02.489418 22680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055 [sys.catalog]: This master's current role is: LEADER
I20260430 20:04:02.490079 22677 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:02.476508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.10.46.62:37697)
0430 20:04:02.477019 (+ 511us) server_negotiation.cc:207] Beginning negotiation
0430 20:04:02.477029 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 20:04:02.477066 (+ 37us) server_negotiation.cc:408] Connection header received
0430 20:04:02.477777 (+ 711us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:04:02.477803 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:04:02.478170 (+ 367us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:04:02.478443 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:04:02.479543 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:04:02.480377 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:04:02.481293 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:04:02.481588 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:04:02.484471 (+ 2883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:04:02.484520 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:04:02.484539 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:04:02.484602 (+ 63us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:04:02.486811 (+ 2209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:04:02.487340 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:04:02.487349 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:04:02.487362 (+ 13us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:04:02.487442 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:04:02.487688 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:04:02.487693 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:04:02.487695 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:04:02.488278 (+ 583us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:04:02.488519 (+ 241us) server_negotiation.cc:1036] Waiting for connection context
0430 20:04:02.488835 (+ 316us) server_negotiation.cc:300] Negotiation successful
0430 20:04:02.489152 (+ 317us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":149,"threads_started":1}
I20260430 20:04:02.490528 22674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260430 20:04:02.490622 22688 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 20:04:02.496575 22688 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 20:04:02.503312 22688 catalog_manager.cc:1357] Generated new cluster ID: 11a83eb933f041a5861598943515cd5d
I20260430 20:04:02.503398 22688 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 20:04:02.513266 22688 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 20:04:02.514178 22688 catalog_manager.cc:1514] Loading token signing keys...
I20260430 20:04:02.525044 22688 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 730b3cd235c04dfda4e82dd31bb10055: Generated new TSK 0
I20260430 20:04:02.525712 22688 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.10.46.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:04:03.352118 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.46.20:34783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.1:0
--local_ip_for_outbound_sockets=127.10.46.1
--webserver_interface=127.10.46.1
--webserver_port=0
--tserver_master_addrs=127.10.46.62:37697
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 20:04:03.511669 22733 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:04:03.512136 22733 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:04:03.512290 22733 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:04:03.517397 22733 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:04:03.517577 22733 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:04:03.517763 22733 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.1
I20260430 20:04:03.524369 22733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.46.20:34783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.46.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.1
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:04:03.525825 22733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:04:03.526984 22733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:04:03.535751 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:04:03.535851 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:04:03.535748 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:04:03.538029 22733 server_base.cc:1061] running on GCE node
I20260430 20:04:03.538607 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:04:03.539458 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:04:03.547039 22733 hybrid_clock.cc:648] HybridClock initialized: now 1777579443546955 us; error 101 us; skew 500 ppm
Apr 30 20:04:03 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579443, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:04:03.551616 22733 init.cc:377] Logged in from keytab as kudu/127.10.46.1@KRBTEST.COM (short username kudu)
I20260430 20:04:03.553310 22733 webserver.cc:492] Webserver started at http://127.10.46.1:44531/ using document root <none> and password file <none>
I20260430 20:04:03.554167 22733 fs_manager.cc:362] Metadata directory not provided
I20260430 20:04:03.554304 22733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:04:03.554601 22733 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 20:04:03 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579443, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
I20260430 20:04:03.572983 22733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data/instance:
uuid: "46d5a68477a84d0f913c809ebffe85b7"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "625a4a61c59c729951543ada5be00e50"
server_key_iv: "7a185cacd6e2644ac1a4c96c835f0e9a"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:03.573771 22733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance:
uuid: "46d5a68477a84d0f913c809ebffe85b7"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "625a4a61c59c729951543ada5be00e50"
server_key_iv: "7a185cacd6e2644ac1a4c96c835f0e9a"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:03.578612 22733 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260430 20:04:03.620059 22751 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:03.626153 22733 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260430 20:04:03.626343 22733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal
uuid: "46d5a68477a84d0f913c809ebffe85b7"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "625a4a61c59c729951543ada5be00e50"
server_key_iv: "7a185cacd6e2644ac1a4c96c835f0e9a"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:03.626485 22733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:04:03.660794 22733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:04:03.681516 22733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:04:03.681743 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:04:03.682405 22733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:04:03.683525 22733 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:04:03.683591 22733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:03.683646 22733 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:04:03.683673 22733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:03.707347 22733 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.1:34079
I20260430 20:04:03.708673 22733 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/data/info.pb
I20260430 20:04:03.716079 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 22733
I20260430 20:04:03.716248 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-0/wal/instance
I20260430 20:04:03.716504 10424 external_mini_cluster.cc:1468] Setting key 4870604befb658b37b7e10f071ca247a
I20260430 20:04:03.722889 22866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.1:34079 every 8 connection(s)
Apr 30 20:04:03 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579443, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.1@KRBTEST.COM for kudu/127.10.46.62@KRBTEST.COM
I20260430 20:04:03.733037 22869 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:03.713111 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.10.46.1:55247 (local address 127.10.46.62:37697)
0430 20:04:03.717458 (+ 4347us) server_negotiation.cc:207] Beginning negotiation
0430 20:04:03.717463 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 20:04:03.717476 (+ 13us) server_negotiation.cc:408] Connection header received
0430 20:04:03.717515 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 20:04:03.717517 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 20:04:03.717560 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 20:04:03.717646 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 20:04:03.719838 (+ 2192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:04:03.720691 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 20:04:03.722645 (+ 1954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 20:04:03.722865 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:04:03.725185 (+ 2320us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 20:04:03.725214 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 20:04:03.725218 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 20:04:03.725252 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 20:04:03.729028 (+ 3776us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:04:03.729772 (+ 744us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:04:03.729776 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:04:03.729778 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:04:03.729847 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 20:04:03.730250 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 20:04:03.730254 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 20:04:03.730256 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 20:04:03.730452 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 20:04:03.730548 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 20:04:03.732548 (+ 2000us) server_negotiation.cc:300] Negotiation successful
0430 20:04:03.732686 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4262,"thread_start_us":106,"threads_started":1}
I20260430 20:04:03.733397 22870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:03.713487 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:55247)
0430 20:04:03.714874 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:03.714919 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0430 20:04:03.716554 (+ 1635us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:04:03.718433 (+ 1879us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 20:04:03.718447 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 20:04:03.718855 (+ 408us) client_negotiation.cc:190] Negotiated authn=SASL
0430 20:04:03.719588 (+ 733us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:04:03.719608 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:04:03.721325 (+ 1717us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 20:04:03.721330 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 20:04:03.721996 (+ 666us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 20:04:03.722008 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 20:04:03.722247 (+ 239us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 20:04:03.723067 (+ 820us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 20:04:03.723094 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0430 20:04:03.724954 (+ 1860us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 20:04:03.729235 (+ 4281us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:04:03.729243 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:04:03.729259 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:04:03.729634 (+ 375us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:04:03.729978 (+ 344us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 20:04:03.729982 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 20:04:03.729984 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 20:04:03.730133 (+ 149us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 20:04:03.730608 (+ 475us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 20:04:03.730615 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 20:04:03.730942 (+ 327us) client_negotiation.cc:770] Sending connection context
0430 20:04:03.731191 (+ 249us) client_negotiation.cc:241] Negotiation successful
0430 20:04:03.731445 (+ 254us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":117,"threads_started":1}
I20260430 20:04:03.734760 22867 heartbeater.cc:344] Connected to a master server at 127.10.46.62:37697
I20260430 20:04:03.735078 22867 heartbeater.cc:461] Registering TS with master...
I20260430 20:04:03.735819 22867 heartbeater.cc:507] Master 127.10.46.62:37697 requested a full tablet report, sending...
I20260430 20:04:03.737640 22638 ts_manager.cc:194] Registered new tserver with Master: 46d5a68477a84d0f913c809ebffe85b7 (127.10.46.1:34079)
I20260430 20:04:03.739456 22638 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.46.1@KRBTEST.COM'} at 127.10.46.1:55247
WARNING: no policy specified for kudu/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.46.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.10.46.62 from keytab /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.10.46.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.46.2@KRBTEST.COM" created.
I20260430 20:04:03.828245 22725 server.cc:273] Received an EOF from the subprocess
F20260430 20:04:03.829722 22698 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.10.46.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab.
I20260430 20:04:03.832165 10424 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
/tmp/dist-test-taskqmdCRr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.46.20:34783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.46.2:0
--local_ip_for_outbound_sockets=127.10.46.2
--webserver_interface=127.10.46.2
--webserver_port=0
--tserver_master_addrs=127.10.46.62:37697
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.10.46.20:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7f5e94acedcd google::LogMessage::Fail() at ??:0
@ 0x7f5e94ad2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5e94ace7cc google::LogMessage::Flush() at ??:0
@ 0x7f5e94acff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5e968e7157 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f5e968e3f59 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f5e968e7e85 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5585cfc65244 std::function<>::operator()() at ??:0
@ 0x7f5e95fd37ea kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5e969176db start_thread at ??:0
@ 0x7f5e93c8561f clone at ??:0
W20260430 20:04:03.952502 22876 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 20:04:03.952790 22876 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 20:04:03.952888 22876 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 20:04:03.956547 22876 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 20:04:03.956652 22876 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 20:04:03.956771 22876 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.46.2
I20260430 20:04:03.961688 22876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.46.20:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.46.20:34783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/krb5kdc/kudu.keytab
--principal=kudu/127.10.46.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.46.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.46.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.10.46.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.46.2
--log_dir=/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11680
I20260430 20:04:03.962859 22876 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 20:04:03.963745 22876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 20:04:03.970806 22895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:04:03.970790 22893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 20:04:03.970790 22892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 20:04:03.971406 22876 server_base.cc:1061] running on GCE node
I20260430 20:04:03.972010 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 20:04:03.972756 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 20:04:03.974041 22876 hybrid_clock.cc:648] HybridClock initialized: now 1777579443973982 us; error 71 us; skew 500 ppm
Apr 30 20:04:03 dist-test-slave-0bdw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579443, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 20:04:03.977288 22876 init.cc:377] Logged in from keytab as kudu/127.10.46.2@KRBTEST.COM (short username kudu)
I20260430 20:04:03.978470 22876 webserver.cc:492] Webserver started at http://127.10.46.2:37503/ using document root <none> and password file <none>
I20260430 20:04:03.979064 22876 fs_manager.cc:362] Metadata directory not provided
I20260430 20:04:03.979115 22876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 20:04:03.979327 22876 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 20:04:03 dist-test-slave-0bdw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777579443, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.46.2@KRBTEST.COM for HTTP/127.10.46.20@KRBTEST.COM
I20260430 20:04:03.997026 22876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data/instance:
uuid: "4172fbe907da4bf08698b61787e455f2"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "eac3133c4e25009c414e88ac5d8dcf3b"
server_key_iv: "3f1df87e604d4c2af4eaeb3e8e9e1f09"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:03.997697 22876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance:
uuid: "4172fbe907da4bf08698b61787e455f2"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "eac3133c4e25009c414e88ac5d8dcf3b"
server_key_iv: "3f1df87e604d4c2af4eaeb3e8e9e1f09"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:04.002583 22876 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260430 20:04:04.020771 22903 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:04.022676 22876 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 20:04:04.022852 22876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal
uuid: "4172fbe907da4bf08698b61787e455f2"
format_stamp: "Formatted at 2026-04-30 20:04:03 on dist-test-slave-0bdw"
server_key: "eac3133c4e25009c414e88ac5d8dcf3b"
server_key_iv: "3f1df87e604d4c2af4eaeb3e8e9e1f09"
server_key_version: "kuduclusterkey@0"
I20260430 20:04:04.023001 22876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 20:04:04.051080 22876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 20:04:04.053901 22876 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 20:04:04.054126 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 20:04:04.054731 22876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 20:04:04.055742 22876 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 20:04:04.055866 22876 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:04.055934 22876 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 20:04:04.055969 22876 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 20:04:04.067507 22876 rpc_server.cc:307] RPC server started. Bound to: 127.10.46.2:45165
I20260430 20:04:04.067523 23016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.46.2:45165 every 8 connection(s)
I20260430 20:04:04.068572 22876 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/data/info.pb
I20260430 20:04:04.071527 10424 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqmdCRr/build/debug/bin/kudu as pid 22876
I20260430 20:04:04.071653 10424 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqmdCRr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777579284342945-10424-0/minicluster-data/ts-1/wal/instance
I20260430 20:04:04.071985 10424 external_mini_cluster.cc:1468] Setting key c0e93916640f2ab66b64a28677a7e511
W20260430 20:04:04.111505 21882 connection.cc:570] client connection to 127.10.46.62:37697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 20:04:04.112581 23019 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 20:04:04.071028 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:36947)
0430 20:04:04.072251 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:04.072297 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 20:04:04.073317 (+ 1020us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 20:04:04.111908 (+ 38591us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":142,"threads_started":1}
W20260430 20:04:04.113159 23017 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:37697 (0 consecutive failures): Network error: Failed to ping master at 127.10.46.62:37697: Client connection negotiation failed: client connection to 127.10.46.62:37697: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260430 20:04:04.114010 23019 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:04.113700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:35035)
0430 20:04:04.113866 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:04.113933 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260430 20:04:04.114715 23019 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:04.114469 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:33883)
0430 20:04:04.114599 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:04.114648 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260430 20:04:04.114924 23017 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260430 20:04:04.742558 22867 heartbeater.cc:499] Master 127.10.46.62:37697 was elected leader, sending a full tablet report...
I20260430 20:04:04.743640 23021 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:04.743145 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:56987)
0430 20:04:04.743456 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:04.743524 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
W20260430 20:04:04.743993 22867 heartbeater.cc:646] Failed to heartbeat to 127.10.46.62:37697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
I20260430 20:04:05.116250 23022 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:05.115656 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:48771)
0430 20:04:05.116031 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:05.116108 (+ 77us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":168,"threads_started":1}
I20260430 20:04:05.745270 23023 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:05.744749 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:39877)
0430 20:04:05.745090 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:05.745157 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
I20260430 20:04:06.117626 23024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:06.117161 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:37633)
0430 20:04:06.117448 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:06.117506 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260430 20:04:06.746789 23025 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:06.746286 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:47431)
0430 20:04:06.746617 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:06.746678 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260430 20:04:07.119088 23026 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:07.118588 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:45161)
0430 20:04:07.118930 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:07.118985 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260430 20:04:07.748286 23027 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:07.747731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:36561)
0430 20:04:07.748115 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:07.748175 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":150,"threads_started":1}
I20260430 20:04:08.120646 23028 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:08.120182 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:47009)
0430 20:04:08.120482 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:08.120541 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260430 20:04:08.749938 23029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:08.749335 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.1:60767)
0430 20:04:08.749755 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:08.749818 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":88,"threads_started":1}
I20260430 20:04:09.122627 23030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 20:04:09.122135 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.10.46.62:37697 (local address 127.10.46.2:49621)
0430 20:04:09.122463 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0430 20:04:09.122520 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.46.62:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":124,"threads_started":1}
F20260430 20:04:09.159948 10424 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1777579449 (unix time) try "date -d @1777579449" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000028b8) received by PID 10424 (TID 0x7f7d6a70cd80) from PID 10424; stack trace: ***
@ 0x7f7d7639e980 (unknown) at ??:0
@ 0x7f7d739f0e87 gsignal at ??:0
@ 0x7f7d739f27f1 abort at ??:0
@ 0x7f7d74c5ddcd google::LogMessage::Fail() at ??:0
@ 0x7f7d74c61b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7d74c5d7cc google::LogMessage::Flush() at ??:0
@ 0x7f7d74c5ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7d779e043b kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f7d779dd156 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55a85ce9d7bf kudu::SecurityITest::StartCluster() at ??:0
@ 0x55a85ce85622 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f7d75c690ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7d75c5dbdb testing::Test::Run() at ??:0
@ 0x7f7d75c5dd9d testing::TestInfo::Run() at ??:0
@ 0x7f7d75c5e377 testing::TestSuite::Run() at ??:0
@ 0x7f7d75c5e77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f7d75c6960d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7d75c5de63 testing::UnitTest::Run() at ??:0
@ 0x55a85ced5b0e RUN_ALL_TESTS() at ??:0
@ 0x55a85ced4cfb main at ??:0
@ 0x7f7d739d3c87 __libc_start_main at ??:0
@ 0x55a85ce6cefa _start at ??:0
2026-04-30 20:04:09.386 UTC [22584] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 20:04:09.388 UTC [22431] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 20:04:09.388 UTC [22434] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 20:04:09.388 UTC [22430] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 20:04:09.388 UTC [22435] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 20:04:09.390 UTC [22432] FATAL: terminating connection due to unexpected postmaster exit